Service 'index' exited with status 134

Hi,
I run Couchbase 7.1.0 on Ubuntu 20.04. The following errors occurs:

Service 'index' exited with status 134. Restarting. Messages:
2022-06-27T13:08:13.287+00:00 [Info] RebalanceServiceManager::rebalanceJanitor Running Periodic Cleanup
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7f80db219a80)
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 15031182372585180544 FlushTime 173ns CommitTime 3.160754ms TotalFlushTime 1.4252ms TotalCommitTime 5m43.882682393s
2022-06-27T13:08:14.897+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 15031182372585180544 Creating New Snapshot SnapshotInfo: seqnos: 223, 0, 69883 committed:true
2022-06-27T13:08:14.898+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 15031182372585180544 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 223, 0, 69883 committed:true) SnapType DISK_SNAP SnapAligned true SnapCreateDur 39.957733ms SnapOpenDur 134.563µs
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp before terminating.

I have two nodes and 200 documents. This error is causing com.couchbase.client.core.error.InternalServerFailureException: Internal Couchbase Server error in my java app sometimes.

CC @sduvuru for forestdb related question.

Looks like some memory corruption. Can you run forestdb_dump on the forestdb file and see if the disk image looks ok?

forestdb_dump:

2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 0). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.545+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 1). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 38, SB No.: 2). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Failed to read the superblock: incorrect block marker (marker: 0, SB No.: 3). Note: this message might be a false alarm if upgrade is running.
2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: 2064 non-block aligned bytes discarded from a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'

2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp'

2022-07-04T18:32:30.546+00:00 [ERRO][FDB] Error in READ on a database file '/opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp', errno = 2: 'No such file or directory'

Unable to open /opt/couchbase/var/lib/couchbase/crash/9662c678-598a-4096-e71bbdab-5c747394.dmp

another crash:

Service 'index' exited with status 134. Restarting. Messages:
2022-07-06T15:24:43.968+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 17532055274852714362 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 2, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 14.075703ms SnapOpenDur 55.282µs
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::Commit SliceId 0 IndexInstId 12084728265744008952 FlushTime 305ns CommitTime 194.802µs TotalFlushTime 0s TotalCommitTime 17.987216926s
2022-07-06T15:24:43.981+00:00 [Info] ForestDBSlice::OpenSnapshot SliceId 0 IndexInstId 12084728265744008952 Creating New Snapshot SnapshotInfo: seqnos: 1, 0, 146036 committed:true
2022-07-06T15:24:43.981+00:00 [Info] StorageMgr::handleCreateSnapshot Added New Snapshot Index: 12084728265744008952 PartitionId: 0 SliceId: 0 Crc64: 833398431925170828 (SnapshotInfo: seqnos: 1, 0, 146036 committed:true) SnapType FORCE_COMMIT SnapAligned true SnapCreateDur 12.338927ms SnapOpenDur 31.423µs
assertion failed [hbmeta.prefix != NULL] at /home/couchbase/jenkins/workspace/couchbase-server-unix/forestdb/src/hbtrie.cc:1267 ((nil) != 0x7fecfcb4f490)
Breakpad caught a crash in forestdb. Writing crash dump to /opt/couchbase/var/lib/couchbase/crash/94f72b8b-717a-4e23-e39eac87-213f8425.dmp before terminating.

I think this bug was introduced in couchbase 7.x

Is there any solution to the problem? There is always at least one crash per day.
It’s impossible to downgrade (7.1 → 6.6) for us because of scopes / collections.