Couchbase Auto-Compaction consequences to set it to only during the night?

We have an older version of Couchbase on a Windows Server:
Version: 4.6.5-4742 Enterprise Edition (build-4742)

We are saving our app session there and the connector is old so that is why we haven’t been able to Upgrade to a newer version.

We had to upgrade recently our infrastructure saldy the people mainting the previous infraestructure quit and we had no metrics of how the Previous Couchbase Server was performing and what IOPs limits it had.
So when we started to hit big loads we started to get a lot of issues with our session and our issues not being able to login or being kicked. I haven’t worked much with Couchbase before but reading the limited App logs it seemed the issue was Couchbase. I started to read Couchbase logs but I couldn’t see any problems per se, I just noticed that the system seem to be compacting constantly. And when we were checking our metrics we saw the Hard Drives of the Couchbase Server running at its limite of IOPs. It was so bad that apparently Couchbase couldn’t write some of its logs at times, because there were some time periods where It didn’t log any misses.

So we decided to double its IOPs limit since we had no information of how it was performing in the past and what limits it had, but we knew that for a long while the processes were going to be write-heavy since we flushed several buckets that had gigabytes of data when we migrated. And additionally we moved the compaction time to run during the night.

After that we haven’t had the issue yet. But Couchbase seems to be hitting the IOPs limit again. What intrigues me is that I never saw any error anywhere, yet the system couldn’t find the user sessions in couchbase.

Does this behavior sound reasonable to you? Also could there be any bad consequences to having the compaction run only during the night? I am sorry if this questions seem rather basica I am really not usually working in Couchbase tunning and maintenace so I am a bit at a loss here.

Hello Dzyann_Leleur

Thank you for using Couchbase Server. It’s a shame you are unable to upgrade, as many improvements into throughput and rebalance speed has gone into the 5.x, 6.x and 7.x product versions. As an Enterprise Edition user, I would highly suggest you work with your account manager and get a sizing exercise performed, to ensure you are configured to meet the requirements of your use-case. If you’re unsure who your account manager is, an email to sales@couchbase.com can get you started .

In terms of your question around having compaction running only at night, the downsides to doing this is that you will potentially need a lot more disk space to contain all of your mutations during the day until the compaction process is allowed to run later on. This can also affect performance as the the large fragmented vBucket files are not as efficient as smaller / compacted vBuckets. One other thing to look out for is making sure you have allocated enough time to actually complete the compaction process across all of your data-set. You will need to do some trial and error to see if your compaction is able to complete in the allotted time. Some methods to speed up compaction include adding additional resources such as disk IO or whatever else is the limiting factor such as CPU. And another factor that can help is adding additional Couchbase Server Nodes, as this lets more vBuckets get compacted in parallel.

Thanks,
Ian McCloy (Principal Product Manager, Couchbase)