We have some data that we record every second (or thereabouts) which adds up to a lot of data over the course of a day. We store the data as a protobuf attachment, because JSON has a fair amount of overhead for small but extremely frequent data. When we update the attachment frequently, Couchbase’s footprint seems to grow very quickly until we compact – I imagine it is keeping multiple attachment versions and then pruning them at compact time.
If the iOS version of the application is asked to compact while the app is running in the background, bad things seem to happen (or, at least, they used to in older versions of Couchbase). You seem to be able to work around those bad things if you start/stop the sync and use custom NSDataWritingOptions, but … we want to keep the data protected.
Ultimately, we want to save this information as a protobuf file containing the day’s data to persistent storage frequently and we want it to be in Couchbase either immediately or with some lowered frequency throughout the day. So we’re considering options like storing this data in the filesystem temporarily and only passing it off to Couchbase periodically. This allows us to save the information regularly, and update Couchbase periodically so that compaction and background operation is less of a problem for us. This adds a lot of complexity and reduces the recency of our data on the server, so if I can avoid it, I’d be happy to do so.
We’ve also upgraded recently to Couchbase 1.2 and with that means we could use ForestDB, which seems to have auto-compaction features, although I haven’t investigated that option much.
This leads me to questions:
- Does anyone have any high-level suggestions for options I may not have considered?
- Are any of my premises obviously wrong? For instance:
- Is there a way that I can store this information as a Couchbase attachment and not have problems with the disk space growing significantly between compact runs?
- Or is there a way to allow compacting the database on iOS (or at least the attachments) in the background without touching NSDataWritingOptions
- Do ForestDB’s auto-compaction features help me? Does it work with attachments?
Thanks.