Large Attachments in Sync Gateway

We have a unique scenario that drove us to Couchbase Mobile; our tablet application will be without network connectivity 75+% of the day and we need to have all of our data available at all times cached on the device.We have a total of around 100gb of video, pictures, and other data. I had hoped to store everything as attachments and have it sync from server to device whenever network connectivity was available.

As I was reading through the mobile documentation I found information about how attachments are stored on the file system and not directly inside the database and therefore they could be arbitrarily large. However I’ve now run into a problem where putting a 250mb video file to the Sync Gateway is silently failing. I get a return that says success but then the attachment isn’t there. Can the Sync Gateway handle syncing these large files that Couchbase Lite stores as attachments?

There is currently a limit of 20mb per attachment. Someone can correct me if I’m wrong, but I believe this reflects a 20mb limit in Couchbase Server.

I found that 20mb limit too, after the fact. Are there alternative options for storage at all? I found CBFS which I’m playing with now but it looks pretty unmaintained. https://github.com/couchbaselabs/cbfs. Given my end goal I really don’t want to put all the content into an S3 bucket and manage the sync myself. But I fear that is what I’ll have to do: store the location in couchbase and then manually sync the files.

How often are you changing/moving the files around? You can chunk them. Here’s also some food for thought (sorry, I think the title is a bit unfortunate, but I didn’t write it): http://blog.couchbase.com/2016/january/large-objects-in-a-database

Thank you. I am exploring chunking as well. Our client is extremely concerned about chunking’s performance so I’m not sure if that will work strictly from the perspective of making them comfortable with our solution.

@jason.burich Just had to pipe up to say Chunking FTW!

I have a slightly different scenario where there was a need to have a pretty small file that I needed to continually append to, which adds up to a lot of data transfer.

Instead, I went with immutable, multiple attachments and never looked back. In theory, there shouldn’t be much of a hit from using multiple attachments versus a single one… However, I don’t know how long it takes to chunk and combine your videos.