Huge allocation of "Other Data" in Memory Cache Stats


#1

Hello -

We are currently performing testing with a two-code cluster of Azure A3 VMs with 7GB of memory per-node. We have the per-server ram quota set to the suggested amount of 4182MB. Everything works fine, except that rebalance operations seem to take a VERY long time, even with a small number of documents (roughly 250K docs @ 500MB total size).

One concern is that possibly the VMs are running short of memory during the rebalance. Looking at the “Memory Cache” section, it says that only 400MB are “In Use” but then 6.3GB is consumed by Other Data. Does that sound right, and is that something we should be concerned with?

I’m not sure how to gauge whether the cluster is experiencing memory pressure during the rebalance operation, but I want to make sure I understand what the “Other Data” mem calc means, and whether it’s something we need to be concerned about.

Any links to general memory management for Couchbase would be helpful as well, I did quite a bit of searching, and there isn’t much out there that describes how to determine if Couchbase is “healthy” from a memory perspective.

Thanks,

  • Jeff

#2

Hi, Jeff,

Which graph do you see 6.3GB for other data?
Is that on cluster overview page or detail graph under server nodes tab?
Is that possible that you can post screenshot here?

Here are some doc on how CB manages memory:
http://docs.couchbase.com/admin/admin/Concepts/concept-workingset-mgmt.html

Thanks,
Qi


#3

Hi Qi -

I see it for my individual nodes, and I see it in both Dev and Production. Everything starts out much lower, but eventually the “Other Data” creeps all the way up to consuming almost all the system memory. Screenshot below, does that seem right to you? Ideally we would be able to review a counter in the UI or on the linux console that would give us indicators when Couchbase becomes memory starved.

Thank you for the link on the memory management docs, I’ll read that over in detail.


#4

Thanks, Jeff.

In UI, “Other Data” refers to memory consumed by all other Couchbase Server processes (beam.smp, memcached, etc.) other processes, and that used by the operating system for disk cache and so on.
Doc: http://docs.couchbase.com/admin/admin/UI/ui-cluster-summary.html

In CB 4.0, we did a few improvement in memory management and reduce size of memory RSS. So in 4.0, you may see a great reduction in memory RSS.

I haven’t seen anything like this but let me find out more for you.

Thanks,
Qi


#5

Other Data is (as qicb mentioned) is “RAM used by other processes on the server”, i.e. excluding Couchbase.
I suggest you use your favourite OS monitoring application (top, Resource Monitor, etc) to examine that what memory is being consumed by.


#6

Thanks guys. From what I’ve gathered using other system resource monitoring tools, The “Other Data” calc is either inaccurate or is including the disk cached pages. From what I’ve read online, most system monitoring tools ignore memory used by the disk cache because it does not compete with the memory used by processes.

I believe this suspicion was confirmed by reviewing the “RAM Usage” at the summary level which reads “30%” in the screenshot below. This percentage matches the memory consumption I’m seeing in other monitoring tools. I personally think it would make more sense for Memory Cache graphic to report the same value as the node summary. That would be much less confusing.