Is there an attachment limit in CB Lite, Sync Gateway, or CB Server?

Hey all, we’re testing a new architecture and I’m concerned about one possible problem. We’re currently concurrently updating attachments with incremental data on multiple devices.

This isn’t a huge problem, but it means that we’re really flooding our network requests. E.g. if I’m updating an attachment with 1kB every minute (for example), that means I’m synching 1k, then 2k, then 3k then 4k - for only 4k of raw data -> This can add up over time.

An alternative method is that we create a new attachment for each incremental amount of data. So, instead of 1 attachment of 4k in the example above (updated 4 times), we have 4 attachments of 1k each (far nicer for networking and concurrency).

So, million dollar question is… Is there a limit to how many attachments I can safely have in CBL on iOS and Android, or in Sync Gateway, or in Couchbase? Like… If I have 20 attachments totaling 1MB, is that a problem?

Thanks!
-SJ

No, there’s no limit on the number of attachments per doc, and there’s not much overhead. This is definitely a better way to go than appending data to a single attachment, since we don’t support delta-compression in the replicator yet.

Thanks Jens!

Delta compression would be amazing! There isn’t much overhead though? That surprises me, as shouldn’t that add a lot of latency if we now add dozens of attachment requests?

I did an investigation/prototype of delta compression last year, but it hasn’t made it into the feature list yet. It’s definitely something many customers want, though.

By “not much overhead” I mean that the cost is basically O(n) as you’d expect, not O(n^2) as when you append to a single attachment. For example a document with 200 1k attachments is going to take about twice as long to replicate as one with 100 1k attachments. The entire doc gets sent as a single HTTP body in MIME multipart format, with each attachment one body part. If only some of the attachments have changed, only those are included. So if you keep adding attachments one at a time, each replication only includes the new attachment.

(The document JSON body does get sent every time, and it contains metadata about all the attachments in the _attachments property, so there’s a slight O(n^2) effect there, but it hopefully shouldn’t be a problem.)

Ah, okay, perfect. For some reason, I assumed that each attachment would be it’s own REST request or something similar, so that could ramp up on network connections.

Good news though. I’m going to test this new architecture, so here’s hoping!

HTTP requests don’t equate to connections. Clients sent multiple requests over the same socket.

Thanks! Exactly what I want to hear!

:+1:

I recommend the book High Performance Browser Networking if you want to learn more about how HTTP works and performs. (It’s a more general book than the title implies. It’s definitely not just about browsers, or even just about HTTP.)

Hi @jens ,

I would like to add my question to this thread.

In my current situation when I attach a 2MB+ image to a document in device A having continuous syncing, it does not sync with device B and when I try with low-quality image it does
My question is that Is there a limit on the size of the attachments.

2MB should easily be supported. As stated in our docs, Couchbase Lite supports attachments of unlimited size, although the Sync Gateway currently imposes a 20MB limit for attachments synced to it. So 2MB is well within the limits. I believe 20MB is also the max size of each document.

But you indicated

So how much is “+” really?

Hi @priya.rajagopal

attachment size is not more than 5MB

here is the stack trace i am getting:
com.couchbase.lite.replicator.RemoteMultipartRequest {PUT, my Sync Gateway url}: executeRequest() Exception: java.net.SocketException: Connection reset. url: my sync gateway url
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:192)
at java.net.SocketInputStream.read(SocketInputStream.java:120)
at okio.Okio$2.read(Okio.java:139)
at okio.AsyncTimeout$2.read(AsyncTimeout.java:237)
at okio.RealBufferedSource.indexOf(RealBufferedSource.java:345)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:217)
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:211)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:75)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
at okhttp3.interna

This happens when the connection between device and sync gateway breaks. There can be many reasons for it like network, proxy used in front of sync gateway.

If you are using NginX/ELB then please look into following

Basically this is not an error but noise which says that the connection is not stable.