I’m having an issue where Couchbase is returning stale data only first thing in the morning. Let me explain…
I’m running production Couchbase servers (4.6) with 3 nodes. In this scenario, I have 2 web requests that are hit by a client app in succession:
- endpoint that (over)writes a couchbase document with data pulled from an external source
- endpoint that reads the content of the document that was provided in the response from the first endpoint
So, there are 2 separate requests on the web server cluster, each request is likely to hit a different webserver that will connect to my couchbase cluster.
First thing in the morning, the 2nd endpoint will return old data. If I run the process over and over again I can not reproduce this behavior, as the first endpoint updates the data, the second one reads the updated data as expected. It’s just the first one in the morning (so say after 12 hours of no activity) that I see the problem.
I should also note that in the first endpoint hit, after writing the data, I read it back and log the contents, which are correct. So the 1st endpoint is working properly, and seems to be writing the data properly. This is consistent even on the first attempt of the morning. However the 2nd endpoint still shows the stale data. This specific functionality is to transfer an end user’s data when they move periodically. I have to be able to rely on the durability of the data on the first try after long periods of inactivity.
I have tried passing the replace_to = -1 and persist_to = -1 (a PHP Couchbase lib comment says that “-1” = all active nodes)
Any help would be much appreciated. Thanks