Write intensive compaction time


#1

I have three questions,

  1. How much space needed for compaction process.
  2. How to set auto-compaction based on specific interval.
  3. How to set auto-compaction based on actual disk usage and not fragmentation.

My scenario:

I have a cluster of 3 Couchbase servers with the following spec for each server: 4 vCPU, 8 GB memory, 30 GB disk space.

We use Couchbase as our caching layer with data updated intensively. Making our disk usage expand so quickly. Currently we are writing about 3GB per minute. Doc Fragmentation always between 75-90%. We will revise how our caching work later.

For now, We set up crontab to trigger compaction every 15 minutes, which already used up about 70% of the disk. Which sort of work but I don’t want to rely on external crontab to keep the system running. Is there a way to make auto-compact run every 15 minutes? This 15 minute interval is our safety magic number. We tried to compact when disk is almost full and ran to scenario where disk already full before compaction completed and froze up Couchbase. Is this the right approach?

Total Disk Size


#2

Hi,

Thanks for using Couchbase. Can you please let me know which version of CB are you using?

Regarding compaction, here are a few guidelines:

  1. In the worse case scenario, you may need have a disk space of 2X or 3X of your data size in order for compaction to complete. This is because CB uses append only file format. In the worse case scenario, both old and new file coexist and they both require large disk space.
  2. Yes, you can set compaction strategy based on interval or fragmentation:
    http://docs.couchbase.com/admin/admin/Tasks/tasks-autocompact-strategies.html
    http://docs.couchbase.com/admin/admin/Tasks/tasks-compaction-process.html
    Unfortunately, we don’t support triggering compaction based on disk usage at this time.

I don’t think triggering compaction when your disk is almost full is good idea. This is because during compaction, extra disk space is needed for hold both old and new files. Therefore, I would recommend compaction starts running much earlier, for example, when disk is half full or even earlier to add buffer.

Thanks,
Qi