How to check that all documents of a bucket are persisted


#1

Hello

My server needs execute some “global” views to extract some information at start up, and the views index must be up to date at this time. So I need ensure that all documents are persisted to the disk that I can
execute the views with stale=false.

The libcouchbase supplies the Observe method that can check the persisted state of an individual document, so I wonder if there exists a graceful method to check the persisted state of the whole bucket.

Or is it a correct solution to check if the disk write queue is empty(according to the web console : ep_queue_size+ep_flusher_todo) from the result of /pools/default/buckets/{bucket_name}/stats?

Any suggestions would be appreciated, thanks.


#2

Hello,

So you have 2 things you can do:

  • use the observe as you stated, but you need to key that you want to look at.
  • you can also look at the disk write queue, if this queue is empty this means you do not have any write, so documents/values are on disk

Very often, when people are asking me about the “index consistency”, I answer what I just did but also add the following question/comment to the discussion:

  • Do you really need to have a consistent index/query for this part of the application?

I am doing this just to be sure that developers are asking themselves the question and avoid doing too much in there application. Quite often this is not a mandatory thing, and the application can deal with the “eventual consistency” of the index , either because the application does not need “all the data all the time” or simply by using another approach (for example use another document as cache) to deal with the data…

Regards
Tug
@tgrall


#3

thanks a lot:)

The index/query consistent need is often from the business perspective, such as the rank list of all the players of a MMO game world(the business department require its consistent so that they can rewards the players accordingly). It would be too large to use an individual document as the cache. So I need the consistence just at the server start up, and caches the rank list in the server’s memory in the lifetime. In the period of the server runs, the index/query consistence is needless cause it’s consistent in the server memory.

thanks again, regards.