How "strictly" are temporary documents stored in buckets?

#1

On Couchbase server 2.5.1 we store some temporary document items, with one-day expiry (.NET struct TimeSpan(1,0,0,0)). These items tend to have 10-20-minute update intervals so they have sliding expiration. When the external party in charge of the item fails to sent an update within an extended duration, we forcefully delete the item anyway and re-create another one.

For most part this perpetual update mechanism works fine. Once in a while, however, we observe the activity log the full list of items (obtained through a view) is less than expected. The document items just seem to “evaporate” from the bucket/view.

Since the update/refresh mechanisms are all happening in less than a day, it seems unlikely the document items would have crossed their expiry time. (To be safe at the moment the expiry is set to two days now). Is there something about time-limited items that might cause them to expiry from the bucket much quicker than set?

Occasionally we also observe the view dropping to a lesser count than actual document item count, and then “recovering” to full count on next check. What explains this fluctuating count in the view?

Null string value on retrieving documents from bucket view
#2

Hi @icelava, when you query the view, which stale option are you using? false, ok, or update_after.
If you need view maintenance to be fast under higher consistency (stale=false), I’d highly recommend going to 3.0. the view maintenance in 3.0 is done in memory so you get fresher indexes and lower latency for changes to show up in the index.
thanks
-cihan

#3

I am unfamiliar with stale options; simply grab the view and get the items.

public IEnumerable<TEntity> GetView()
{
	var view = _CouchBaseClient.GetView(_ViewName, _ViewName);
	var items = view.Select(r => r.GetItem());
	return items.Select(i => JsonConvert.DeserializeObject<TEntity>(i as string));
}

I am not so worried about the view taking a while for new items to make it into the list. What I am wondering about is if temporary document items get ejected from the bucket earlier than expected; thus the view also losing that which had been deleted.

#4

thanks for the clarification. The view or the bucket won’t prematurely expire items. However I think what you are experiencing is a delayed snapshot on the view. If you want to check thing items by item, make sure to run the query with GetView(…).Stale(StaleMode.False) so you can be sure that all thing persisted up to the point of the query is visible in the view in 2.5 - to ensure something is persisted, you need to run the set operation with PersistTo flag set to 1. without this stale option, you could be getting whatever is available in the view. That means partial results from the view which could explain the randomness you are seeing.
Note that In the 3.0 version anything in memory will also be visible with just StaleMode.False flag so you won’t need the persistTo flag.
thanks
-cihan

#5

Ok if the item expiry settings are honoured then it is a mystery why the items are being “evaporated” from the bucket.

From our activity log we can observe like within a 41-minute duration (definitely shorter than a day), the item count drops from 6 to 5. And this is not about the inconsistency of the view; the item is gone in the bucket, so we don’t expect the view to have it. Our application code to forcefully remove unwanted items hasn’t kicked in yet.

In fact, I think this explains the problem with Null string value on retrieving documents from bucket view when some times the view ends up with underlying null items - the item has disappeared from the bucket but has yet to sync itself according to the change.

#6

Think we’ve found the culprit. The “phantom” process removing items was found to be residing on a separate rogue web server operating an older version of code; that server should not be executing the module.

Patched up that server to disable the module, and hopefully no more mysterious disappearances!

#7

Good to hear. Thanks @icelava