About couchbase ep-engine on memory management and eventually persistant


#1

I have just read some ep-engine code, any one know and could help to answer couple of questions?:

1.looks there is no garbage collection, since all allocation and deallocation are new/delete items, how could we handle memory fragmentation.

2.ep means eventually engine, but the name stuck, which write asyn to disk, when server down, those data setted in memcache which has not already write to disk will lose, right?


#2

For #1, it’s true that there is no garbage collection but ep-engine does use shared pointers so items that are pointed to will have their destructor called when fully dereferenced. I don’t quite know what memory fragmentation issue you’re referring to, but in most deployments ep-engine uses TCMalloc, and the Couchbase team have worked with the project there and even contributed to it after testing to ensure it meets their needs.

For #2, it is true that items which are in memory and not persisted are at risk of loss, however… First, know that items are scheduled to be persisted immediately, so the time to persist is only bound by how fast the underlying storage can receive it. Second, know that all Couchbase official clients implement durability requirements (as covered in the javadoc, for example) which allow you to have the application block until the item is persisted.


#3

thanks for the quick reply

1.garbage fragmentation means there are a lot of small holes which failed to fulfill new comparablly large memory requirement, eg, memory usage is 50% percent and lot of holes that we could not new for a large set.
and i failed to find tcmalloc link in ep-engine makefile

  1. I don’t think it provide persisted set, since server side return after set to memory, then add to disk queue and write to disk async, which means server down, data in disk queue lossed.

#4

I don’t think you’ll find the tc-malloc link in the ep-engine makefile because it’s loaded by the memcached process which then loads ep-engine.

I don’t understand the next part of your reply. It is true that things received which are not persisted could be lost, but that’s why SDKs provide durability requirements like the PersistTo or ReplicateTo argument in Java/.NET.