'Longs' are converted to scientific notation when replicated to a shadow bucket

Mobile
1

We’re working on an Android Mobile application using Couchbase Mobile and have hit a problem when using ‘shadow buckets’ for mobile synchronisation. I was wondering if anyone else had encountered the problem, if it was a known issue - any information at all on this would be very useful. The problem is as follows:

When an attribute with a large ‘long’ value (a little more precisely: an integer value of size greater than {say} a Java int primitive, i.e. > 2^31-1) is specified in a document, then Couchbase stores this field without using scientific notation in the application bucket. We have ascertained this by examining the operating system files used by Couchbase. However, when a document containing a ‘long’ is copied from the application bucket to a corresponding shadow bucket, then the ‘long’ will be stored using scientific notation. Again, we have ascertained this by examining the files used by Couchbase.

As an example of the problem: if I were to store the current Unix epoch time (1425295827909) in an attribute called ‘t_now’ in a Couchbase document, then the attribute will be stored as:

“t_now” : 1425295827909

in the application bucket but as:

“t_now” : 1.425295827909e12

In the shadow bucket.

Note that this problem is not apparent when viewing documents in the Couchbase server admin console, as conversion from scientific notation to ‘plain’ (non-scientific) format appears to take place here.

This unexpected conversion to scientific notation is causing a range of problems for us. In particular, when we create a doc with a ‘long’ at Couchbase server, this ‘long’ will be converted to scientific notation on copying to the shadow bucket. This value will then be synced over to all mobile devices and then, at some stage, back to the server. The nett result of all this is that, eventually, any ‘long’ in ‘plain’ (non-scientific) notation at the server will be overwritten with the same value expressed using scientific notation.

IMO this is a bug as it seems to me that attribute values stored in a document in a shadow bucket should be exactly the same (in terms of value and representation) as the corresponding values stored in the document in the application bucket.

2

This is a general issue when processing JSON — there’s potential loss of fidelity when numbers are read into platform numeric types and then later written back out. We’ve made some attempts to avoid it in SG, but at the time most of SG was being written, the Go language’s JSON parser didn’t have the full ability to preserve numeric values exactly as written. It does now (via the json.Number type.)

(The reason you don’t see this issue with Couchbase Server itself is that it never rewrites JSON. The JSON data your client writes to a document is just written literally as bytes. (Documents don’t even have to contain JSON, they can be arbitrary blobs.) The only part of Couchbase Server that interprets documents as JSON is the view engine, and it only reads, never writes.)

Could you please file this as an issue on Github? The fix is conceptually straightforward — make sure JSON numbers are parsed as the Number type not the default float64 — but will probably require a lot of little changes around the codebase in places where it reads numbers.

1 Like
3

Thanks for the info. I’ve raised the issue on GitHub here - https://github.com/couchbase/sync_gateway/issues/694.