CBL Replication issue on attachments

Mobile Couchbase Lite
1

Using CouchbaseLite/2.1.2 (.NET; Microsoft Windows 10.0.18362 ) Build/13 LiteCore/ (15) Commit/9aebf28

I can successfully add a blob to my _attachment element and store the data in CBL.

However replication throws the following error
[Sync] ERROR: {Push#1} Got error response to rev 1fc6c5f7-81ec-4b0a-8a40-0dc23aa55a18 2-c0d9082a8fe6875caf1a95baa0d6562cbb9c8d39 (seq #3848): HTTP 400 ‘Missing data of attachment “profile_image”’

My corresponding json is as follows:
“_attachments”: {
“profile_image”: {
“content_type”: string; // ‘image/jpeg’, ‘image/png’, ‘video/3gpp’, etc…
“digest”: string;
“length”: number;
“revpos”: number;
“stub”: boolean;
}
},

I have seen replication errors mentioned with older versions but not lately.
What should I look for to fix this?

Sample code:
var attachments = new MutableDictionaryObject();
if (resident.ImageBytes != null)
{
attachments.SetBlob(“profile_image”, new Blob(“image/jpeg”, resident.ImageBytes));
}
doc.SetDictionary("_attachments", attachments);

2

This sounds familiar; it might be a Sync Gateway bug that was fixed. (@bbrks?) What version of SG are you running? I’d recommend updating to the latest mobile software, v 2.6. We’ve fixed a number of replication issues since 2.1.

3

I think this might be due to the way the blob is being stored under the _attachments property. I don’t think this is generally something we support. The blob should be stored elsewhere in the document.

That error is only returned by Sync Gateway when it attempts to store an attachment digest without "stub": true. Looks like that bit of code hasn’t changed in the last couple of years. It sounds plausible that the special legacy attachments handling could cause problems here, if the blob is then changing the properties inside _attachments.

Either way, it would be useful to get a packet capture so we can take a look at the raw rev message being sent over the wire.

4

Will reproduce and get a copy of the message being sent up to sync gateway along with our version

5

When you say blobs stored under _attachments property not something generally supported you mean you think it will work if we store it under a user-named element like “profile_image”?

Here is the log file info from our server (V2.1.0-121 they say)
2019-09-30T14:21:51.400Z [INF] SyncMsg: [79fd54c3] #8: Type:proposeChanges #Changes: 1 User:HG30QQ2
2019-09-30T14:21:51.736Z [INF] SyncMsg: [79fd54c3] #9: Type:rev Id:306174e9-59b7-4784-9a8a-d0cc92485c35 Rev:1-3f0286bd212b3a7a6fb88a95a4cb1ccf73687f94 Sequence:4168 User:HG30QQ2
2019-09-30T14:21:54.613Z [INF] SyncMsg: [79fd54c3] #9: Type:rev --> 400 Missing data of attachment “profile_image” Time:2.877618965s User:HG30QQ2
2019-09-30T14:21:55.100Z [INF] SyncMsg: [79fd54c3] #10: Type:setCheckpoint Client:cp-Kv6+pz5BEz/t+fRuYF9cVNsIya0= Rev:0-91 User:HG30QQ2
2019-09-30T14:24:02.728Z [INF] WS: [79fd54c3] Error: receiveLoop exiting with WebSocket error: read tcp 10.3.2.180:4984->10.3.1.160:55684: read: connection reset by peer
2019-09-30T14:24:02.728Z [INF] WS: [79fd54c3] BLIP/Websocket Handler exited: read tcp 10.3.2.180:4984->10.3.1.160:55684: read: connection reset by peer
2019-09-30T14:24:02.728Z [INF] HTTP: [79fd54c3] #13298: --> BLIP+WebSocket connection error: read tcp 10.3.2.180:4984->10.3.1.160:55684: read: connection reset by peer
2019-09-30T14:24:02.728Z [INF] Changes: MultiChangesFeed done (to HG30QQ2)

6

Technically you are not allowed to use property names starting with “_” at the top level of a document; that’s reserved for metadata. But “_attachments” is an odd case because in CouchDB, and thus in CBL 1.x and in SG, that’s where attachments went. But in CBL 2 you can put attachments anywhere you want using the Blob API.

7

So we changed our C# code to set the blobs into the root of our document, not the _attachments, and it works fine.
However our portal application is not seeing the blog as part of the _attachments dictionary as expected.
We have an android app which handles blobs the same way (I used it as a sample for our C# app) and the record it send to the portal seem to have the blob saved under the _attachments element.

Any ideas why the android sync would work differently from the windows sync?

8

Adding some details on what we are seeing now. In our C# code we add the blob to the document:

       doc.SetBlob(ProfileImage, resident.ImageBytes != null ? new Blob("image/png", resident.ImageBytes) : null);

        Db.Save(doc);
        doc.Dispose();

The json for the resulting document in CB shows this:

{
    "_attachments": {
        "blob_/profile_image": {
            "content_type": "image/png",
            "digest": "sha1-VGwgUcU3b+aysS0THCSTzVSg0Hg=",
            "length": 180101,
            "revpos": 2,
            "stub": true
        }
    },
    "_id": "158d0b04-7f02-4dff-827a-eb4f882a6ac5",
. etc....
"profile_image": {
        "@type": "blob",
        "content_type": "image/png",
        "digest": "sha1-VGwgUcU3b+aysS0THCSTzVSg0Hg=",
        "length": 180101
    },

etc…

However when we try to retrieve the image:
https://our-server.com/account_data/158d0b04-7f02-4dff-827a-eb4f882a6ac5/profile_image

{
    "error": "not_found",
    "reason": "missing attachment profile_image"
}

So the json is formed correctly but the image does not seem to make it to CB

Any ideas?

9

I think there is another issue. For example I have a website where you can do reservation and online check in and update profile photo. Now I have a feature that I can change data on my mobile even when I am offline. So I will need to provide the image at CBL and I am using CBL 2.6 along with other profile json data.

Now the point is if I want to add image using sync gateway I only have 1 option and thats using sync gateway rest api provided at https://docs.couchbase.com/sync-gateway/current/rest-api.html#/attachment.

The image will be saved as attachment and cbl 2.x using blob. I know auto covenversion Is done. Need answer on following questions:

  1. Is there a blob api for sync gateway. To directly add images as blob.
  2. How much overhead is on cbl for conversion. And is this conversation happens everytime we open the app?
  3. If there is no blob api do we know when it will be available?

Regards
Pankaj

10

Pankaj,
I think you hit the problem. Talking with our Couchbase guys they just asked whether this is a new record or an edit. And that you have to update blobs separately from the rest of the records. Clunky certainly, will do the code changes and hopefully they will work.

11

I separated the update of a blob from the update of the text portion of the document. Still the same results,
the attachment does not get updated on the server side. Here is the code for image updating,
What am i missing?

 public static void SetImage(string residentId, string imageName, byte[] image)
    {
        var doc = Db.GetDocument(residentId)?.ToMutable(); 
        if (doc == null)
        {
            return;
        }

        var attachments = doc.GetDictionary(Attachments);
        if (attachments != null)
        {
            attachments.Remove(imageName);
            attachments.Remove("blob_/" + imageName);
        }

        if (image != null)
        {
            doc.SetBlob(imageName, new Blob("image/png", image));
        }

        Db.Save(doc);
        doc.Dispose();
    }
12

However when we try to retrieve the image:
https://our-server.com/account_data/158d0b04-7f02-4dff-827a-eb4f882a6ac5/profile_image

That’s not the correct URL. The name of the attachment is blob/profile_image, not profile_image. This URL should work: https://our-server.com/account_data/158d0b04-7f02-4dff-827a-eb4f882a6ac5/blob%2Fprofile_image

13

Sync Gateway only recognizes blobs in the root-level _attachments dictionary. You can directly add images using the REST API you linked to.

How much overhead is on cbl for conversion. And is this conversation happens everytime we open the app?

The replicator just adjusts the JSON slightly to put the metadata in the _attachments dictionary it sends to SG. And when pulling a doc it moves metadata from _attachments to the correct path (if the revision was created by CBL.) There’s no noticeable overhead for this.

If there is no blob api do we know when it will be available?

I don’t know. The Sync Gateway engineers want to do it, but so far there have been higher priority things to do first.

14

You don’t have to, though it’s easier.

You can PUT the entire document at once including blobs, but you either have to add the base64-encoded data as a data property in the attachment, or send the document in MIME multipart/related format with the attachment as a separate nested body.

15

Sorry, same error as before, tried that as well last week.

16

I was able to confirm (with both a Xamarin iOS and native iOS app ) that the attachments created on Couchbase Lite via setBlob API can be retrieved via the attachments REST endpoint .
It appears that you may not have specified the blob name correctly in the REST API (it needs to be appropriately URL encoded)

Example (Setting blob on Couchbase Lite side):

// imageData is byte[] 
mutableDocument.SetBlob("imageData", new Blob("image/jpeg", userProfile.ImageData));

REST API:

curl -X GET \
  'http://localhost:4985/userprofile/user::user1/blob_%2fimageData?rev=7-ef634ef5a85a94120509f809df961c2c5e745125' \
  -H 'Accept: application/json' \
  -H 'Cache-Control: no-cache'
17

Thanks Priya, am in the process of using that api in my class. just some debugging to do…

18

seems like to set the blob you also need the rev of the document. Having trouble getting that from my VS environment. Nothing ic on SO and others working as of yet

19

I am confused. I thought you were setting your blob via Couchbase Lite. In any case, If you are using the REST API, then yes- every PUT that is a doc update (regardless of whether it’s a attachment or any document body update) needs the revId to be specified. Its described in the API specs.
I would also recommend that you walk through one of our getting started / beginner “user profile” tutorials- they demonstrate use of blobs.

20

Even with the rev and API our blob updates are not working. The tutorials are nice but have nothing about blobs in them and handling of them seems to be separate from the text portions of the document.

Im attaching our current code to see if anyone can spot the problem:
our bucket is account_data and the public url for the api is contained in cfg.CouchbaseSyncUrl
The dictionary header contains authorization info which we use in GET and PUT operations
Revisions are being pulled correctly…

private static async void PutImage(string docId, byte[] imageBytes, string imageName, Dictionary<string, string> 
header)
    {
        var cfg = AppServices.Container.Resolve<Config>();
        // get latest revision from ResidentQuery
        var couchbaseUrl = cfg.CouchbaseSyncUrl.AbsoluteUri.Replace("wss", "https");
        var baseUrl = $"{couchbaseUrl}account_data/{docId}";
        var requestTask = NetworkUtil.GetJsonAsync(new Uri(baseUrl), header).ConfigureAwait(true);
        JObject jObject = requestTask.GetAwaiter().GetResult();
        var revision = jObject.GetValue("_rev")?.ToString();
        if (string.IsNullOrWhiteSpace(revision) || imageBytes == null)
        {
            return;
        }

        string json = Convert.ToBase64String(imageBytes);

        var url = $"{baseUrl}/blob_%2f{imageName}?_rev={revision}";

       var responseMessage = await NetworkUtil.SendJsonRequestAsync("PUT", new Uri(url), header, json).ConfigureAwait(true);
    }