How to find the setting for indexer.plasma.backIndex.evictSweepInterval

Couchbase Server
1

I ran the GET request as below
http://127.0.0.1:9102/settings, but it did not list the setting for indexer.plasma.backIndex.evictSweepInterval.

How can I find the value for this parameter

2

@rajib761 From indexing/secondary/common/config.go the setting is 300 seconds in most recent code:

	"indexer.plasma.backIndex.evictSweepInterval": ConfigValue{
		300,
		"Time interval to sweep through all pages in an index (in sec)",
		300,
		false, // mutable
		false, // case-insensitive
	},
3

Thanks Kevin, I also wanted to know even if the periodic eviction runs every 300 secs, until the RAM memory quota is hit, it will not actually evict the index to disk. Is my understanding correct? I just ran a test and saw this behavior but just wanted to be doubly sure.

4

@rajib761 I had thought index data would not get evicted until the Index services bumps up against its assigned memory quota, but @varun.velamuri corrected me below. Sorry for the prior innaccurate info (removed in this edit).

5

@rajib761

Periodic eviction can happen even if the memory used is not above the memory quota. As the blog (Using Eviction Effectively to Manage Memory in Couchbase GSI )says,

When the memory used is either over the quota or above a preset minimum threshold, the pages not in the working set are evicted by the swapper thread.

1 Like
6

@varun.velamuri Where is this minimum threshold defined? Is the evictMinThreshold the minimum threshold?

If I understand this correctly, then if my memory quota is 100 GB in a node with 128 GB RAM, as soon as the index data reached 101 GB, the periodic eviction will kick in and start evicting the index pages to disk. Am I right?

"indexer.plasma.mainIndex.evictMinThreshold": ConfigValue{
	0.5,
	"Minimum memory use for periodic eviction to run. When memory use over min threshold," +
		" eviction will not run if plasma estimates all indexes can fit into quota.",
	0.5,
	false, // mutable
	false, // case-insensitive
},
"indexer.plasma.mainIndex.evictMaxThreshold": ConfigValue{
	0.9,
	"Maximum memory use for periodic eviction to run.  Once reach max threshold," +
		" periodic eviction will run regardless if it is going to run into DGM or not.",
	0.9,
	false, // mutable
	false, // case-insensitive
},
7

I stand corrected – I did not think pages actually got evicted until the memory was needed.

8

Where is this minimum threshold defined?

You have pointed out the correct config setting

Not exactly. Once memory used exceeds 50G, then plasma will try to estimate if the indexed data can fit into 100G. If it thinks all the data requires less than 100G, then periodic eviction will not evict pages from memory. If plasma estimates that indexed data requires more than 100G, then periodic eviction will evict pages from memory.

If the memory used goes beyond 90G, then periodic eviction will definitely run.

9

@varun.velamuri
Hi Varun
One more thing I noticed is that when we ingest data, the index RAM usage is almost double the index data size. Below are some metrics that I have captured. Is this normal? Where does the RAM gets used?

Index Data Size RAM usage Index RAM/Data
22 GB 46 GB 2.1
47 GB 97 GB 2.1
57 GB 110 GB 2.0
10

@rajib761 What is the version of couchbase-server you are using? We have changed the computation of data_size stat from server v6.5. Earlier versions of server report incorrect data_size.

Also, can you capture the output of api/v1/stats endpoint (Index Statistics API | Couchbase Docs) and share it here. The output of this endpoint will help to account the memory.

Thanks,
Varun

11

We are using 6.6.2. Let me capture the stats endpoint output

12

@varun.velamuri
Hi Varun, it took some time to get the index stats. Below I am sharing. We have 4 index nodes

Node name: xxxxxxxxx.net

cpu_utilization: 9.2
index_not_found_errcount: 9.2
indexer_state: Active
memory_free: 109553229824
memory_quota: 128849018880
memory_rss: 15145664512
memory_total: 135028908032
memory_total_storage: 14950608896
memory_used: 15141107712
memory_used_queue: 0
memory_used_storage: 13610666961
needs_restart: False
num_connections: 8
num_cpu_core: 8
storage_mode: plasma
timestamp: 1631842834368484212
timings_stats_response: 60524 127787245570 296384833859196628
uptime: 9h21m30.880368723s

Node name: xxxxxxxxx.net

cpu_utilization: 5.366666666666667
index_not_found_errcount: 5.366666666666667
indexer_state: Active
memory_free: 121033568256
memory_quota: 128849018880
memory_rss: 2863398912
memory_total: 135028908032
memory_total_storage: 2819497984
memory_used: 2949126144
memory_used_queue: 0
memory_used_storage: 2537963166
needs_restart: False
num_connections: 11
num_cpu_core: 8
storage_mode: plasma
timestamp: 1631842834419158087
timings_stats_response: 64462 39646985836 33064446560989160
uptime: 10h8m45.71665982s

Node name: xxxxxxxxx.net

cpu_utilization: 6.9331022299256695
index_not_found_errcount: 6.9331022299256695
indexer_state: Active
memory_free: 128304955392
memory_quota: 128849018880
memory_rss: 665649152
memory_total: 135028908032
memory_total_storage: 454025216
memory_used: 611495936
memory_used_queue: 0
memory_used_storage: 378157355
needs_restart: False
num_connections: 7
num_cpu_core: 8
storage_mode: plasma
timestamp: 1631842834459989301
timings_stats_response: 64460 135058117888 320175844275875304
uptime: 10h8m35.148244661s

Node name: xxxxxxxxx.net

cpu_utilization: 4.633333333333333
index_not_found_errcount: 4.633333333333333
indexer_state: Active
memory_free: 26392973312
memory_quota: 128849018880
memory_rss: 74042077184
memory_total: 135028908032
memory_total_storage: 78165524480
memory_used: 78295343104
memory_used_queue: 0
memory_used_storage: 71211473607
needs_restart: False
num_connections: 10
num_cpu_core: 8
storage_mode: plasma
timestamp: 1631842834509990862
timings_stats_response: 60521 34673833760 26598673165417046
uptime: 9h21m27.889244507s

13

@rajib761 Thanks for sharing the stats. From the stats, most of the contribution to RSS is coming from storage (memory_total_storage is very close to memory_rss). I am assuming that you are using Enterprise Edition of couchbase-server with standard global secondary storage mode (i.e. plasma storage engine)

From 6.5, the index data_size for plasma indexes is computed as the product of : (a) Compressed data on disk (b) Compression ratio. This gives an estimate of the amount of data that is actually maintained by the storage layer if it were to be in memory. You might be seeing a huge difference between RSS and data_size as the compression ratio (or) compressed data on disk can vary when there is on-going workload.

Do you see a large difference when the indexes are stable i.e. with few or no mutations?

Thanks,
Varun

14

@varun.velamuri Plasma is listed as the mode in the provided stats:

15

@varun.velamuri
Hi Varun, i will try to get the statistics when there are no mutations. We are using enterprise version 6.6.2

If I understand you correctly, you are saying that memory_total_storage is not the actual size of the index data. It needs to be multiplied with the compression ratio get the actual index data size. if that is the case, how can i get the compression ratio to calculate the memory data size. Also, is it like when we are doing a lot of queries, the index data is uncompressed and brought back to the memory and that is when it is hitting the memory quota limit leading to eviction

Updates based on my further investigation

Looks like the default compression ratio is 3. Am I right? So, if I have memory_total_storage is 90GB, actual index size will be 270GB and I will need that amount of RAM if i want 100% index memory residency. Also what is a back index?

			// data_size is the total key size of index, including back index.
			// data_size for MOI is 0.
			if dataSize, ok := GetIndexStat(index, "data_size", statsMap, true, clusterVersion); ok {
				index.ActualDataSize = uint64(dataSize.(float64))
				// From 6.5, data_size stat contains uncompressed data size for plasma
				// So, no need to consider the compressed version of data so that other
				// parameters like ActualKeySize, AvgDocKeySize, AvgSecKeySize does not break
				if indexerVersion >= common.INDEXER_65_VERSION {
					if index.StorageMode == common.PlasmaDB {
						if config["indexer.plasma.useCompression"].Bool() {
							// factor in compression estimation (compression ratio defaulted to 3)
							index.ActualDataSize = index.ActualDataSize / 3
						}
					}
				}
			}

			// memory usage per index
			if memUsed, ok := GetIndexStat(index, "memory_used", statsMap, true, clusterVersion); ok {
				index.ActualMemStats = uint64(memUsed.(float64))
				index.ActualMemUsage = index.ActualMemStats
				totalIndexMemUsed += index.ActualMemUsage
			} else {
				// calibrate memory usage based on resident percent
				// ActualMemUsage will be rewritten later
				index.ActualMemUsage = index.ActualDataSize * index.ActualResidentPercent / 100
				// factor in compression estimation (compression ratio defaulted to 3)
				if index.StorageMode == common.PlasmaDB {
					if config["indexer.plasma.useCompression"].Bool() {
						index.ActualMemUsage = index.ActualMemUsage * 3
					}
				}
				totalIndexMemUsed += index.ActualMemUsage
			}
16

@rajib761 ,

If I understand you correctly, you are saying that memory_total_storage is not the actual size of the index data

yes. memory_total_storage is not the total amount of indexed data. It is the total amount of memory used by storage later. E.g., If there is 100G of indexed data and memory_quota is 50G, then memory_total_storage can be 45G and the remaining 55G of index data is persisted on disk in compressed form.

It needs to be multiplied with the compression ratio get the actual index data size.

We do not multiply memory_total_storage with compression ratio as the indexed data in memory is already uncompressed. We get the data_size from disk (which is in compressed form) and them multiply it with compression ratio.

if that is the case, how can i get the compression ratio to calculate the memory data size

We have not exposed compression_ratio as the data_size is contains the uncompressed value. Just having compression ratio will not help as we will also require the size of compressed data on disk

Also, is it like when we are doing a lot of queries, the index data is uncompressed and brought back to the memory and that is when it is hitting the memory quota limit leading to eviction

Yes. If scan requires data from disk, it will recover index data into memory, uncompress it and evict existing pages in memory to disk.

Looks like the default compression ratio is 3. Am I right?

No. The actual compression ratio can be larger or smaller depending on the data and the amount of data that can be compressed. The code you are looking at will use compression_ratio as “3” for the sake of estimation.

Also what is a back index?

For each index, GSI maintains two storage instances. Main index and back-index. Main index contains all the indexed keys in sorted order. Back-index contains a mapping between docID and the indexed data. Back-index is maintained to solve below cases:

a. Consider “doc1” having indexed field {“a”: “123”}
b. It gets stored as “123, doc1” in MainIndex (Just a representation. Not the actual value in storage)
c. Now, if the field “a” in “doc1” is updated to “345”, then how do we find what was the previous value indexed for this document? (Data service would not send the previous value of the indexed field for performance reasons). We need to find the previous value because we need to delete “123, d0c1” and insert “345, doc1”

To answer (c), there are two ways:

  1. Search the entire main index each time and find the keys with “doc1” suffix → This is not at all optimal
  2. Maintain a mapping between document and the indexed data → This is back-index

When there is a update mutation, indexer will first check the back-index to find the current indexed field, deletes it and then updates the new value.

Thanks,
Varun

17

@varun.velamuri

Thanks a lot Varun for this detailed explanation. So if I had to interpret the below stats

cpu_utilization: 4.633333333333333
index_not_found_errcount: 4.633333333333333
indexer_state: Active
memory_free: 26392973312
memory_quota: 128849018880
memory_rss: 74042077184
memory_total: 135028908032
memory_total_storage: 78165524480
memory_used: 78295343104
memory_used_queue: 0
memory_used_storage: 71211473607
needs_restart: False
num_connections: 10
num_cpu_core: 8
storage_mode: plasma
timestamp: 1631842834509990862
timings_stats_response: 60521 34673833760 26598673165417046
uptime: 9h21m27.889244507s

I will do it as below
memory_rss is the total amount of index data in actual RAM + probably the jemalloc stuff
memory_total_storage is the total amount of index data currently in RAM. Should not this be < memory_rss?
memory_used_storage - This is the amount of storage used by the index data(which is not in RAM and compressed) + back index + any overhead
My memory resident ratio for the index will then depend on how much data I have in memory_total_storage or is it memory_rss?

18

@rajib761

memory_rss: Total amount of memory used by indexer process. This is golang’s memory + JEMalloc’s memory. Indexer maintains some in-memory data structures for book-keeping and mutation processing before giving it to storage. Golang’s memory will include memory allocated to these components. During mutation processing, the golang contribution towards memory can grow and then comes down once mutation processing finishes. This is all buffers + back index + all overheads. memory_rss includes memory_total_storage

memory_total_storage: Total amount of memory used by JEMalloc. Includes JEMalloc fragmentation as well. This includes back_index and main index data that is in memory

memory_used_storage: Actual memory used by JEMalloc avoiding fragmentation i.e. memory_total_storage - JEMalloc fragmentation

My memory resident ratio for the index will then depend on how much data I have in memory_total_storage or is it memory_rss?

It depends on memory_quota actually. If indexed data is more than memory_quota, some data will be evicted to disk

19

@varun.velamuri

If indexed data is more than memory_quota, some data will be evicted to disk

This is where I am having an issue, since I do not know the compression ratio, how do I calculate the index data size to size the RAM accordingly

Also, if memory_total_storage is the index data in RAM(roughly), which metrics gives me the value what is the size of data in disk

20

how do I calculate the index data size to size the RAM accordingly

The data_size stat of each index already represents the uncompressed data. You need not require compression ratio.

which metrics gives me the value what is the size of data in disk

There is a disk_size stat that represents the total index size on disk.