Hi Team,
We faced a Couchbase server down issue in production.
Couchbase community edition is 4.0.0 running on a Red Hat Enterprise Linux Server. Couchbase is setup as a a single node server with 1 Couchbase bucket and 1 memcached bucket with No replication.
The hits to the Couchbase server was very minimal during the issue time and all other system parameters like CPU, RAM and Disk were found as normal. The couchbase service was showing as active but the connections from client got timedout.
We found the below entries in the memcached.log file (Issue time :2021-08-03T03:45:02.368228+09:00 )
But there is no clue on the reason behind this shutdown. Can someone please help to understand the issue behind this shutdown. Thanks in advance.
<<memcached.log>>
2021-08-02T18:19:56.194512+09:00 WARNING (hb_cache) Warning: failed to fetch data from database, vBucket=107 key=cache:cache:0001|APP|2021-12-02|2021-12-06|2||||1|1||||21.279|-157.829||||AQUAALOHASURFWAIKIKI| error=document not found [none]
2021-08-03T02:00:11.087147+09:00 Message repeated 2 times
2021-08-03T03:45:02.368228+09:00 WARNING (hb_cache) Shutting down tap connections!
2021-08-03T03:45:02.368275+09:00 WARNING (hb_cache) Shutting down dcp connections!
2021-08-03T03:45:02.368288+09:00 WARNING (hb_cache) Clean up "eq_dcpq:mapreduce_view: hb_cache _design/cache (prod/main)"
2021-08-03T03:45:02.370876+09:00 WARNING (hb_cache) Had to wait 1063 usec for shutdown
2021-08-03T03:45:02.371968+09:00 WARNING (hb_cache) Had to wait 1062 usec for shutdown
2021-08-03T03:45:02.373061+09:00 WARNING (hb_cache) Had to wait 1065 usec for shutdown
2021-08-03T03:45:02.374146+09:00 WARNING (hb_cache) Had to wait 1061 usec for shutdown
2021-08-03T03:45:02.374165+09:00 WARNING (No Engine) Unregistering last bucket hb_cache