Hi,
What’s the procedure to associate a blob with a document in couchbase-lite-core?
Are there tests in the framework that show how to do that?
Thanks in advance!
Regards,
Nuno
We’d much prefer you use Couchbase Lite for C; I hope you’re porting your code to that.
But if you need this to work in the short term: In LiteCore you save the blob using the c4BlobStore API, then add a dictionary to the document with the metadata, including "@type":"blob" and the required digest property. The revision also needs the ‘has blob’ flag, but I can’t remember whether that’s added automatically or not. It wouldn’t hurt to set it yourself.
@jens I have been on the link you have sent me the other day and it seems that Couchbase Lite for C is still incomplete. Does it support replication through sync-gateway already? By looking at the checkboxes in the README apparently, it didn’t.
What you are saying, matches with what I saw in Doxygen for couchbase-lite-core. I just wanted to have some code to guide me through, such as tests. I saw tests for blob creation but not yet for blob attachment to documents. Maybe didn’t look correctly yet.
It supports replication, but it is less failure-tolerant than Couchbase Lite for {Java, .NET, Obj-C} — it won’t retry if the connection fails for some transient reason, or if the connection goes down. But the same is true of LiteCore. The logic for retry (and mobility / offline support) is currently located in Couchbase Lite for {Java, .NET, Obj-C}, partly because the APIs to get notified of network changes are platform-specific.
So one of the CBL-C tasks is to move that logic down to the LiteCore level where CBL_C can take advantage of it, and to somehow deal with the mobility APIs in a cross platform way.
TL;DR: In either CBL_C or LiteCore, the replicator will work but will fail immediately if it can’t open a connection, or will stop immediately if the connection goes down. (Even if it’s supposed to be a continuous replication.)
@jens before digging into a new C API I need to consolidate my knowledge about all this with my current working project.
I’m trying to add an attachment to a document. I’m using c4Test.cc as a reference (method addDocWithAttachments). What I’m doing in my test app is like this:
QByteArray ba;
ba.append("a");
ba.append("b");
ba.append("c");
C4Error error;
C4BlobKey key;
c4blob_create(_c4BlobStore, QSlice::c4FromQByteArray(ba), nullptr, &key, &error);
QString keyStr = QSlStringResult(c4blob_keyToString(key));
IC4Document* document = nullptr;
C4Error c4err;
QJsonObject data;
QJsonObject attached;
QJsonArray attachments;
QJsonObject attachment;
data["channels"] = QJsonArray() << _channel;
attachment[kC4ObjectTypeProperty] = kC4ObjectType_Blob;
attachment["digest"] = keyStr;
attachment["length"] = ba.size();
attachment["content_type"] = "text/plain";
attachments.append(attachment);
attached["attached"] = attachments;
data["_attachments"] = attached;
QString id = QUuid::createUuid().toString();
qDebug() << "Creating document with id" << id;
QSlString docIdNew = id;
c4::Transaction t(_c4Database);
if (!t.begin(&c4err))
{
logC4Error(QString("Unable to begin transaction for creating new document, DocId = %0").arg(QSlice::qslToQString(docIdNew)), c4err);
exit(-1);
}
QJsonDocument doc = QJsonDocument(data);
QSlResult body = c4db_encodeJSON(_c4Database, (QSlString) doc.toJson(), &c4err);
if (body.isNull())
{
logC4Error(QString("Unable to encode json for new document %1").arg(QSlice::qslToQString(docIdNew)), c4err);
exit(-1);
}
C4DocPutRequest rq = {};
rq.docID = (QSlString) docIdNew;
rq.body = body;
rq.save = true;
rq.existingRevision = false;
rq.revFlags = kRevKeepBody | kRevNew | kRevHasAttachments;
c4::ref<C4Document> docNew = c4doc_put(_c4Database, &rq, nullptr, &c4err);
In the end, the document data is:
{"_attachments":{“attached”:[{"@type":“blob”,“content_type”:“text/plain”,“digest”:“sha1-qZk+NkcGgWq6PiVxeFDCbJzQ2J0=”,“length”:3}]},“channels”:[“ff4fdde2-02fd-41fd-9386-94842f156c5a-drc”]}
However, after the document is create and the replication happens, I can see the following error in the sync-gateway log:
Type:rev --> 400 Invalid attachment Time:1.00262ms User:ff4fdde2-02fd-41fd-9386-94842f156c5a
I’m wondering what is making the attach invalid. If I don’t add an attachment to the document, it is created on the server bucket correctly.
I’m not sure I’m doing everything I need regarding the Blob Store. I’m just doing this:
_c4BlobStore = c4db_getBlobStore(_c4Database, &error);
I’m not making use of c4blob_openStore and I’m not sure if I actually need it.
I’m definitely missing something here. Can you please advise?
Thanks!
Best regards,
Nuno
I have found the answer. I was not formatting the JSON correctly. I had {"_attachments":{“attached”: [ and it should simply be {“attached”: [
Not complaining any more!
Thanks!
To be precise, the problem was that you had the value of _attachments.attached as an array; removing the outermost pair of square brackets would have fixed it.
But the _attachments property is a holdover from CouchDB and has a more strict schema; you should avoid using it, so your current schema is better.
@jens I’m running into a new problem.
I’m able to store a blob into the database. However, it does not seem to be replicated correctly. In the couchbase-server UI I can see a binary document and inside the document data I can see the blob being reference but the length is 0.
When I add the document with the attachment, the replicator breaks with the following error:
18:31:36.947513| [Sync]: {Push#5} Got response for 1 local changes (sequences from 10)
QCBLWebSocket:disconnected "sg-dev.imaginando.pt" "/db/_blipsync" QAbstractSocket::UnknownSocketError
"Unknown error"
18:31:37.291698| [WS] WARNING: {C4SocketImpl#8}==> litecore::repl::C4SocketImpl wss://sg-
dev.imaginando.pt:443/db/_blipsync @0x7ff15d4e21d0
18:31:37.291743| [WS] WARNING: {C4SocketImpl#8} WebSocket closed abnormally with status 1006
18:31:37.300727| [Sync]: {Repl#4} Connection closed with WebSocket status 1006: "Unknown error" (state=2)
18:31:37.301527| [Sync] ERROR: {Repl#4} Got LiteCore error: WebSocket error 1006 "Unknown error"
Replication error occured, aborted "Unknown error"
On the sync gateway side there is this:
sync-gateway_1 | 2019-05-16T17:45:03.008Z [TRC] Bucket: GetRaw("_sync:att:sha1-720tj641+M0XPDyGofv0cNIFbyE=") [363.897µs]
sync-gateway_1 | 2019-05-16T17:45:03.008Z [DBG] Sync+: [590f7ae3] Asking for attachment "blob_/blob" (digest
sha1-720tj641+M0XPDyGofv0cNIFbyE=). User:ff4fdde2-02fd-41fd-9386-94842f156c5a
sync-gateway_1 | 2019-05-16T17:45:03.008Z [DBG] WSFrame+: [590f7ae3] Queued MSG#3
sync-gateway_1 | 2019-05-16T17:45:03.008Z [DBG] WS+: [590f7ae3] Push MSG#3
sync-gateway_1 | 2019-05-16T17:45:03.008Z [DBG] WS+: [590f7ae3] Sending frame: MSG#3 (flags= 0, size= 78)
sync-gateway_1 | 2019-05-16T17:45:03.153Z [DBG] WS+: [590f7ae3] Received frame: RPY#3~ (flags= 1001001,
length=4090)
sync-gateway_1 | 2019-05-16T17:45:03.153Z [INF] WS: [590f7ae3] ERROR decompressing frame: inputLen=4090,
remaining=0, output=1, error=unexpected EOF
sync-gateway_1 | 2019-05-16T17:45:03.154Z [INF] WS: [590f7ae3] Error decompressing frame RPY#3~: unexpected
EOF. Raw frame = <03496200000000ffff1c5a0554dc5ad74d3233998cbb6383438152da525aea46dd5fdddddddd85fa7b757737ead485b6144a694b711d60dc7d2693497ebe7fdd7557b2e
...
At this point, the websocket closes and everything gets corrupted.
I’m wondering if this is the socket that is failing to transmit the data that is asked.
My interpretation is that sync-gateway is asking the blob, the blob data is sent, the data is corrupted and it fails.
Any ideas? I’ve been here for hours without any luck.
Thanks,
Best regards,
Nuno
Yesterday I have spent 8 hours trying to understand why the replicator socket breaks when sync gateway asks for a attachment that I add to a document. I couldn’t figure it out.
To ensure that the problem was not from my side I have made several things:
- Tried another branch from couchbase-lite-core (2.5.0) - No success!
- Tried to disable the SSL websocket layer using civetwebsocketfactory - No success!
- Tried to isolate the operation in a isolated project - No success!
Everytime I add an attachment to a document, it is successfully inserted in the local database but not correctly uploaded to the couchbase server, sync-gateway crashes upon requesting the attachment. The feedback is always the same: Error decompressing frame RPY#3~: unexpected
EOF and connection is closed by peer .
Do you have any insights of what might be happening? I’m sure this is not a couchbase-lite-core problem but something I’m doing wrong. I just can’t seem to find…
Thanks!
Best regards,
Nuno
Are you able to reproduce this in a clean project or create a repro case to show? The SG logs and your other logs don’t match up with the digests and you said you changed the way you are storing them into the documents so it’s likely something else is going on.
- Do the C4Tests pass on your system?
- Can you reproduce the problem without introducing unknown variants into the equation? (QT CBL wrapper)
- Can you write a project that any of us can quickly run and verify?
@borrrden I will try to isolate everything in a C program. I will get back here when I have it.
Can you tell me how to call C4Tests?
Thx!!!
@borrrden test results:
09:29:19.237612| [DB]: Deleting database file /private/var/folders/bw/lynl848s6dg1rfpvsv0frh9m0000gn/T/Litecore_C_Tests/cbl_core_test.cblite2/db.sqlite3 (with -wal and -shm)
All tests passed (392017 assertions in 113 test cases)
I have created a public repo with a test project: https://github.com/imaginando/cb_test
It has already the code to add an attachment together with the document.
but curiously I can’t even create a document and sync it to the db. I can’t find the reason why…
Sync gateway is currently public and allowing guest connections.
What am I missing here?
I have been morning to write this example…
There are two errors in your test project:
- You forgot to encode the JSON (
c4db_encodeJSON) before you tried to save the document - Your JSON is not good because the entire “blob” dictionary is in quotes. Furthermore, the length is also in quotes.
@borrrden thanks for your valuable insights. You were completely right. I have updated the program on github.
I’m now able to set the document on the database but for some reason the attachment doesn’t seem to be recognized because the the property _attachments is empty and I can’t see binary data on the database.
I have triple checked the JSON format and everything appears to be correct. Blocked again… 
Thank you very much for your help!
Best regards,
Nuno
@borrrden I discovered the reason for the attachment not being uploaded. It was a copy/paste typo on the digest. It was not being dynamically set.
I ended up with the problem I was describing earlier. There is the Error decompressing frame error unexpected EOF followed by the dump of the raw frame.
Although the document is created on the database, the digest from the attachment doesn’t match and the length is 0:
{
“attachments": {
"blob/attached/0”: {
“content_type”: “text/plain”,
"digest": “sha1-2jmj7l5rSw0yVb/vlWAYkK/YBwk=”,
"length": 0,
“revpos”: 1,
“stub”: true
}
},
“attached”: [
{
“@type”: “blob”,
“content_type”: “text/plain”,
"digest": “sha1-iypXuV4DbskNO4roX+oGsR+fcik=”,
"length": 95862
}
]
}
I have updated the test again:
I don’t have Qt in the middle now.
Thanks in advance!
Regards,
Nuno
The error in question means that for some reason the content of the frame is not being properly compressed on the client side, or is failing to send the entire thing. I’m not that familiar with this area but maybe @jens has an idea of what is going on. It could be something we take for granted in Lite that I’m not remembering at the moment or it could be some flaw in CivetWebSocket since we don’t use it for actual SG connections in production.
Furthermore I am not able to reproduce this locally. I started up a Sync Gateway on my own machine with several different configurations and versions and your test program worked fine. What is the content of your sync gateway config file?
@borrrden awkward that it doesn’t fail on your side. I hope this is a simple fix because it is preventing me from moving forward…
On the client side I have been using the release/iridium branch
My sync-gateway config file is like this:
{
"logging": {
"console": {
"log_level": "debug",
"log_keys": ["*"]
}
},
"adminInterface":":4985",
"interface": ":4984",
"databases": {
"db": {
"server": "http://couchbase-server:8091",
"bucket": "couchbase_qt",
"username": "DB_USER_NAME",
"password": "DR_USER_PASSWORD",
"enable_shared_bucket_access": true,
"import_docs": true,
"num_index_replicas": 0,
"users": {
"GUEST": {"disabled": false, "admin_channels": ["*"] }
}
}
}
}
Sync-gateway Dockerfile is like this:
FROM couchbase/sync-gateway:2.1.2-community
COPY config.json /etc/sync_gateway/config.json
And docker-compose is like this:
couchbase-server:
image: couchbase
restart: always
labels:
- traefik.backend=cb-dev
- traefik.enable=true
- traefik.frontend.rule=Host:cb-dev.imaginando.pt
- traefik.frontend.entryPoints=http
- traefik.port=8091
volumes:
- couchbase-data:/opt/couchbase/var
sync-gateway:
build: sync-gateway
labels:
- traefik.backend=sg-dev
- traefik.enable=true
- traefik.frontend.rule=Host:sg-dev.imaginando.pt
- traefik.frontend.entryPoints=http
- traefik.port=4984
ports:
- 4985:4985This sounds like a problem in the compression part of the BLIP protocol we use in the replicator (it’s a layer on top of WebSockets), but it’s unclear whether the problem is on the LiteCore or the SG side. It would be great to find a reproducible case. Could you please file an issue against LiteCore, including a link to your test code?