How to store a blob into a document using couchase-lite-core

#9

@borrrden @jens

Yesterday I have spent 8 hours trying to understand why the replicator socket breaks when sync gateway asks for a attachment that I add to a document. I couldn’t figure it out.

To ensure that the problem was not from my side I have made several things:

  • Tried another branch from couchbase-lite-core (2.5.0) - No success!
  • Tried to disable the SSL websocket layer using civetwebsocketfactory - No success!
  • Tried to isolate the operation in a isolated project - No success!

Everytime I add an attachment to a document, it is successfully inserted in the local database but not correctly uploaded to the couchbase server, sync-gateway crashes upon requesting the attachment. The feedback is always the same: Error decompressing frame RPY#3~: unexpected
EOF
and connection is closed by peer .

Do you have any insights of what might be happening? I’m sure this is not a couchbase-lite-core problem but something I’m doing wrong. I just can’t seem to find…

:frowning_face:

Thanks!
Best regards,
Nuno

#10

Are you able to reproduce this in a clean project or create a repro case to show? The SG logs and your other logs don’t match up with the digests and you said you changed the way you are storing them into the documents so it’s likely something else is going on.

  • Do the C4Tests pass on your system?
  • Can you reproduce the problem without introducing unknown variants into the equation? (QT CBL wrapper)
  • Can you write a project that any of us can quickly run and verify?
#11

@borrrden I will try to isolate everything in a C program. I will get back here when I have it.

Can you tell me how to call C4Tests?

Thx!!!

#12

@borrrden test results:

09:29:19.237612| [DB]: Deleting database file /private/var/folders/bw/lynl848s6dg1rfpvsv0frh9m0000gn/T/Litecore_C_Tests/cbl_core_test.cblite2/db.sqlite3 (with -wal and -shm)

All tests passed (392017 assertions in 113 test cases)

#13

@borrrden,

I have created a public repo with a test project: https://github.com/imaginando/cb_test

It has already the code to add an attachment together with the document.

but curiously I can’t even create a document and sync it to the db. I can’t find the reason why…

Sync gateway is currently public and allowing guest connections.

What am I missing here?

I have been morning to write this example…

:frowning:

#14

There are two errors in your test project:

  1. You forgot to encode the JSON (c4db_encodeJSON) before you tried to save the document
  2. Your JSON is not good because the entire “blob” dictionary is in quotes. Furthermore, the length is also in quotes.
#15

@borrrden thanks for your valuable insights. You were completely right. I have updated the program on github.

I’m now able to set the document on the database but for some reason the attachment doesn’t seem to be recognized because the the property _attachments is empty and I can’t see binary data on the database.

I have triple checked the JSON format and everything appears to be correct. Blocked again… :confused:

Thank you very much for your help!

Best regards,

Nuno

#16

@borrrden I discovered the reason for the attachment not being uploaded. It was a copy/paste typo on the digest. It was not being dynamically set.

I ended up with the problem I was describing earlier. There is the Error decompressing frame error unexpected EOF followed by the dump of the raw frame.

Although the document is created on the database, the digest from the attachment doesn’t match and the length is 0:

{
attachments": {
"blob
/attached/0”: {
“content_type”: “text/plain”,
"digest": “sha1-2jmj7l5rSw0yVb/vlWAYkK/YBwk=”,
"length": 0,
“revpos”: 1,
“stub”: true
}
},
“attached”: [
{
@type”: “blob”,
“content_type”: “text/plain”,
"digest": “sha1-iypXuV4DbskNO4roX+oGsR+fcik=”,
"length": 95862
}
]
}

I have updated the test again:

I don’t have Qt in the middle now.

Thanks in advance!

Regards,

Nuno

#17

The error in question means that for some reason the content of the frame is not being properly compressed on the client side, or is failing to send the entire thing. I’m not that familiar with this area but maybe @jens has an idea of what is going on. It could be something we take for granted in Lite that I’m not remembering at the moment or it could be some flaw in CivetWebSocket since we don’t use it for actual SG connections in production.

#18

Furthermore I am not able to reproduce this locally. I started up a Sync Gateway on my own machine with several different configurations and versions and your test program worked fine. What is the content of your sync gateway config file?

#19

@borrrden awkward that it doesn’t fail on your side. I hope this is a simple fix because it is preventing me from moving forward… :frowning:

On the client side I have been using the release/iridium branch

My sync-gateway config file is like this:

{
"logging": {
	"console": {
		"log_level": "debug",
		"log_keys": ["*"]
	}
},
"adminInterface":":4985",
"interface": ":4984",
"databases": {
	"db": {
		"server": "http://couchbase-server:8091",
		"bucket": "couchbase_qt",
		"username": "DB_USER_NAME",
		"password": "DR_USER_PASSWORD",
		"enable_shared_bucket_access": true,
		"import_docs": true,
		"num_index_replicas": 0,
		"users": {
			"GUEST": {"disabled": false, "admin_channels": ["*"] }
		}
	}
}
} 

Sync-gateway Dockerfile is like this:

FROM couchbase/sync-gateway:2.1.2-community

COPY config.json /etc/sync_gateway/config.json

And docker-compose is like this:

couchbase-server:
  image: couchbase
  restart: always
  labels:
    - traefik.backend=cb-dev
    - traefik.enable=true
    - traefik.frontend.rule=Host:cb-dev.imaginando.pt
    - traefik.frontend.entryPoints=http
    - traefik.port=8091
volumes:
  - couchbase-data:/opt/couchbase/var

sync-gateway:
  build: sync-gateway
  labels:
    - traefik.backend=sg-dev
    - traefik.enable=true
    - traefik.frontend.rule=Host:sg-dev.imaginando.pt
    - traefik.frontend.entryPoints=http
    - traefik.port=4984
  ports:
    - 4985:4985
#20

This sounds like a problem in the compression part of the BLIP protocol we use in the replicator (it’s a layer on top of WebSockets), but it’s unclear whether the problem is on the LiteCore or the SG side. It would be great to find a reproducible case. Could you please file an issue against LiteCore, including a link to your test code?

#21

@jens,

I have done that this afternoon. Check couchbase-lite-core issues.

Thanks

#22

@jens @borrrden

I have found the culprit…

Since Jim said it didn’t had any problems running my example locally I decided to do the same. It worked.

So then I have been debugging my whole network architecture do find where the problem laid. It seems that the culprit is Traefik.

I have Traefik to allow me to expose certain urls to the network rather than access via IP address. It seems that Traefik is not handling this case very well. And it seems I’m not the only one complaining:

https://www.google.com/search?q=Error+when+copying+from+client+to+backend%3A+websocket%3A+close+1006&oq=Error+when+copying+from+client+to+backend%3A+websocket%3A+close+1006&aqs=chrome..69i57.1106j0j7&sourceid=chrome&ie=UTF-8

I’m currently using v1.7 of Traefik and using the latest version doesn’t fix the problem. I didn’t try version 2.0 because it has breaking changes and I don’t have time to learn about it right now.

In the mean time, I just wanted to confirm that the bug isn’t in couchbase-lite-core side. I will move forward with my work but I need to find a way of exposing sync-gateway url in a proper way. Any suggestions? Can you ask around inside your team if they know anything about alternatives to do this with Traefik or another kind of proxy service?

Thanks!

Best regards,

Nuno

#23

@sinosoidal Thanks for getting back and confirming that this is not and SG issue!

I don’t know that anyone here is familiar with Traefik. I believe that most of our customers just expose the IP address and close down any unnecessary ports. If you can say more about what you mean by “exposing the url in a proper way”, I can ask around.

#24

@blake.meike sorry if I’m using using the proper language to describe this.

Traefik is a router. Look at this docker-compose snippet to see how it works:

sync-gateway:
 build: sync-gateway
 labels:
   - traefik.backend=sg-dev
   - traefik.enable=true
   - traefik.frontend.rule=Host:sg-dev.imaginando.pt
   - traefik.frontend.entryPoints=http
   - traefik.port=4984 

With Traefik I can say any request to the url sg-dev.imaginando.pt will be forwarded to the docker service sync-gateway at port 4984. This allows me to expose only port 80. Even better is that I can than use Cloudflare to serve that url as https and have a SSL communication. This was working perfectly for documents only. For some reason, it is not working with attachments upload. The websocket is closed with the message:

Error when copying from client to backend: websocket: close 1006

Who’s fault? I don’t know. Uncharted terrain for me! The fact is that without Traefik, it works as it should be.

Maybe around your team of web developers, full stack developers, backend developers and docker wizards, someone knows alternatives to Traefik or even know about this problem that I’m having in details.

Thank you very much,

Best regards,

Nuno

#25

Yeah… I figured that.

This is going to depend on the specifics of Traefik. In order to hold a websocket connection open, a router has to do some clever stateful trickery. I’m not surprised to hear that it fails sometimes.

Lemme ask around.

#26

Hey @sinosoidal,
Just querying around this very forum, I see several other people mentioning Traefik in connection with similar problems.

I think you are probably just going to want to make your SG IP:port visible.

You know, btw, that CBL and the SG support SSL via the wss: URL scheme.

#27

I’m not an easy quitter! :slight_smile:

The use of a reverse proxy such as Traefik makes configuration a breeze and solves a lot of problems.

Just to make this clear, sync-gateway replication works through Traefik for documents. Only attachments are failing. There must be something special in the attachments packets.

Are there any custom headers being transmitted so that data is validated for each transmitted packet? Maybe Traefik is not copying those headers and validation fails.

I have found this -> https://docs.traefik.io/configuration/backends/docker/#custom-headers

Does this ring you any bells?

Thanks!

Best regards,

Nuno

#28

Blob data is not much different than other data: Multi-frame messages. Perhaps this is your issue: