How to find the setting for indexer.plasma.backIndex.evictSweepInterval

I ran the GET request as below
http://127.0.0.1:9102/settings, but it did not list the setting for indexer.plasma.backIndex.evictSweepInterval.

How can I find the value for this parameter

@rajib761 From indexing/secondary/common/config.go the setting is 300 seconds in most recent code:

	"indexer.plasma.backIndex.evictSweepInterval": ConfigValue{
		300,
		"Time interval to sweep through all pages in an index (in sec)",
		300,
		false, // mutable
		false, // case-insensitive
	},

Thanks Kevin, I also wanted to know even if the periodic eviction runs every 300 secs, until the RAM memory quota is hit, it will not actually evict the index to disk. Is my understanding correct? I just ran a test and saw this behavior but just wanted to be doubly sure.

@rajib761 I had thought index data would not get evicted until the Index services bumps up against its assigned memory quota, but @varun.velamuri corrected me below. Sorry for the prior innaccurate info (removed in this edit).

@rajib761

Periodic eviction can happen even if the memory used is not above the memory quota. As the blog (Using Eviction Effectively to Manage Memory in Couchbase GSI )says,

When the memory used is either over the quota or above a preset minimum threshold, the pages not in the working set are evicted by the swapper thread.

1 Like

@varun.velamuri Where is this minimum threshold defined? Is the evictMinThreshold the minimum threshold?

If I understand this correctly, then if my memory quota is 100 GB in a node with 128 GB RAM, as soon as the index data reached 101 GB, the periodic eviction will kick in and start evicting the index pages to disk. Am I right?

"indexer.plasma.mainIndex.evictMinThreshold": ConfigValue{
	0.5,
	"Minimum memory use for periodic eviction to run. When memory use over min threshold," +
		" eviction will not run if plasma estimates all indexes can fit into quota.",
	0.5,
	false, // mutable
	false, // case-insensitive
},
"indexer.plasma.mainIndex.evictMaxThreshold": ConfigValue{
	0.9,
	"Maximum memory use for periodic eviction to run.  Once reach max threshold," +
		" periodic eviction will run regardless if it is going to run into DGM or not.",
	0.9,
	false, // mutable
	false, // case-insensitive
},

I stand corrected – I did not think pages actually got evicted until the memory was needed.

Where is this minimum threshold defined?

You have pointed out the correct config setting

Not exactly. Once memory used exceeds 50G, then plasma will try to estimate if the indexed data can fit into 100G. If it thinks all the data requires less than 100G, then periodic eviction will not evict pages from memory. If plasma estimates that indexed data requires more than 100G, then periodic eviction will evict pages from memory.

If the memory used goes beyond 90G, then periodic eviction will definitely run.

@varun.velamuri
Hi Varun
One more thing I noticed is that when we ingest data, the index RAM usage is almost double the index data size. Below are some metrics that I have captured. Is this normal? Where does the RAM gets used?

Index Data Size RAM usage Index RAM/Data
22 GB 46 GB 2.1
47 GB 97 GB 2.1
57 GB 110 GB 2.0

@rajib761 What is the version of couchbase-server you are using? We have changed the computation of data_size stat from server v6.5. Earlier versions of server report incorrect data_size.

Also, can you capture the output of api/v1/stats endpoint (Index Statistics API | Couchbase Docs) and share it here. The output of this endpoint will help to account the memory.

Thanks,
Varun

We are using 6.6.2. Let me capture the stats endpoint output