Hey we have done an upgrade from couchbase version 5.5.4 to couchbase version 6.5.0 recently.
On one of our Stages the upgrade worked without any errors, but on another stage we had some issues after the upgrade.
We followed performed an rolling online update with the Graceful Failover and Delta Recovery strategy.
Our cluster consists of 3 Nodes with data,query and index service on all 3 nodes.
First Node upgrade worked without any problems. When we upgraded the second Node and tried to delta recovery it we experienced some issues.
First of all a delta recovery with rebalance failed. The Logs says that memcached exited with status 139(log 1) and rebalance failed with reason badmatch,fail. This happened everytime.
After this we couldn’t gracefully autofailover the node, again because of rebalance failures.
Even a restart from the couchbase-server service didn’t help to solve this issue.
Additionally we saw some warnings with “replica with invalid snapshot range.” What is this and what might have caused it? (log 2)
The only solution was to do a full recovery for both remaining nodes.
We haven’t seen any huge memory or CPU usage that might have caused it.
Did someone experienced any similar issue or maybe knows why this happend?
(log 1)
[error_logger:error,2020-04-23T10:52:59.616Z,babysitter_of_ns_1@cb.local:error_logger<0.32.0>:ale_error_logger_handler:do_log:203]** Generic server <0.297.0> terminating
** Last message in was {#Port<0.3956>,{exit_status,139}}
** When Server state == {state,#Port<0.3956>,
{memcached,"/opt/couchbase/bin/memcached",
["-C",
"/opt/couchbase/var/lib/couchbase/config/memcached.json"],
[{env,
[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","5"},
{"CBSASL_PWFILE",
"/opt/couchbase/var/lib/couchbase/isasl.pw"}]},
use_stdio,stderr_to_stdout,exit_status,
stream]},
{ringbuffer,1134,1024,
{[{<<"2020-04-23T10:52:59.458177+00:00 WARNING (SomeBucket2) Slow runtime for 'Warmup - completion' on thread reader_worker_0: 7024 us">>,
125},
{<<"2020-04-23T10:52:59.430425+00:00 WARNING (SomeBucket) Slow runtime for 'Warmup - completion' on thread reader_worker_0: 14 ms">>,
121},
{<<"2020-04-23T10:52:57.917692+00:00 WARNING 50: Slow operation. {\"cid\":\"127.0.0.1:50125/0\",\"duration\":\"697 ms\",\"trace\":\"request=23604958507991445:697766\",\"command\":\"CREATE_BUCKET\",\"peer\":\"127.0.0.1:50125\",\"bucket\":\"\",\"packet\":{\"bodylen\":471,\"cas\":0,\"datatype\":\"raw\",\"extlen\":0,\"key\":\"<ud>SomeBucket</ud>\",\"keylen\":6,\"magic\":\"ClientRequest\",\"opaque\":0,\"opcode\":\"CREATE_BUCKET\",\"vbucket\":0}}">>,
382},
{<<"2020-04-23T10:52:57.900904+00:00 WARNING (SomeBucket) CouchKVStore::initialize: readVBState readVBState:2, name:/opt/couchbase/var/lib/couchbase/data/SomeBucket/862.couch.2">>,
164}],
[{<<"2020-04-23T10:52:57.897474+00:00 WARNING (SomeBucket) CouchKVStore::initialize: readVBState readVBState:2, name:/opt/couchbase/var/lib/couchbase/data/SomeBucket/944.couch.2">>,
164},
{<<"2020-04-23T10:52:57.900894+00:00 WARNING (SomeBucket) CouchKVStore::processVbstateSnapshot vb:862 replica with invalid snapshot range. Found version:1, highSeqno:13, start:14, end:14">>,
178}]}},
undefined,
{ok,{-576460713009,
#Ref<0.138041975.252444673.69369>}},
[<<"2020-04-23T10:52:59.458177+00:00 WARNING (SomeBucket2) Slow runtime for 'Warmup - completion' on thread reader_worker_0: 7024 us">>,
<<"2020-04-23T10:52:59.430425+00:00 WARNING (SomeBucket) Slow runtime for 'Warmup - completion' on thread reader_worker_0: 14 ms">>],
0}
** Reason for termination ==
** {abnormal,139}
(log 2 can also be seen in log 1)
memcached<0.4030.0>: 2020-04-23T10:47:11.220124+00:00 WARNING (someBucket) CouchKVStore::processVbstateSnapshot vb:944 replica with invalid snapshot range. Found version:1, highSeqno:11, start:12, end:12
memcached<0.4030.0>: 2020-04-23T10:47:11.220139+00:00 WARNING (someBucket) CouchKVStore::initialize: readVBState readVBState:2, name:/opt/couchbase/var/lib/couchbase/data/someBucket/944.couch.2
memcached<0.4030.0>: 2020-04-23T10:47:11.221663+00:00 WARNING (someBucket) CouchKVStore::processVbstateSnapshot vb:862 replica with invalid snapshot range. Found version:1, highSeqno:13, start:14, end:14
memcached<0.4030.0>: 2020-04-23T10:47:11.221675+00:00 WARNING (someBucket) CouchKVStore::initialize: readVBState readVBState:2, name:/opt/couchbase/var/lib/couchbase/data/someBucket/862.couch.2