Indices are malformed after service stopped

Hello,
I’m using Community Edition 5.1.1 build 5723 on local env for development and I encounter some annoying issue:

When couchbase server service stops (or computer restarts) it seems like everything ok, all indices are still in place as well as the data but when I run some queries it returns empty results (when before I had results…).
After dropping and building all indices from scratch everything is back to normal.
This happens also when I manually stop the service.
Running on Windows 10.

Beside the fact I can’t turn this service off (and save some resources while not developing) it makes it also any update (or other actions that require restart) to be very time consuming.

Any advice? Thanks.

@dor272,

Can you check the indexer.log for any errors after restart. It would be easy for us to root cause the issue if you can share the indexer logs.

Thanks,
Varun

Hey,
after checking the log I have errors from the following type:

2019-10-30T19:21:24.917+07:00 [Info] CompactionDaemon: Checking fragmentation: IndexInstId: 7288637052112226967 Data:159525, Disk:18415616, ExtraSnapshotData:0, Fragmentation:99.13375148569561%, mode : circular
2019-10-30T19:21:24.917+07:00 [Info] CompactionDaemon: Compacting index instance:7288637052112226967
2019-10-30T19:21:24.926+07:00 [Info] ForestDBSlice::Compact Compacting upto SeqNum 17816. Slice Id 0, IndexInstId 7288637052112226967, IndexDefnId 9242386019601100374
2019-10-30T19:21:24.927+07:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\re_com_dates_7288637052112226967_0.index\data.fdb.12
2019-10-30T19:21:24.932+07:00 [ERRO][FDB] Error in reading the doc length metadata with offset 16003056 from a database file ‘c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\re_com_dates_7288637052112226967_0.index\data.fdb.11’
2019-10-30T19:21:24.933+07:00 [INFO][FDB] Forestdb closed database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\re_com_dates_7288637052112226967_0.index\data.fdb.12
2019-10-30T19:21:24.933+07:00 [Error] CompactionDaemon: Index instance:7288637052112226967 Compaction failed with reason - Unknown forestdb error 16003056

Thanks

@dor272,

From the log snippet, there seems to be an issue with reading ForestDB files after restart. I did not find any known issue around this area for 5.1.1. As a workaround, can you try updating the couchbase-server version to 6.0.3 and let us know your observations. In the meantime, I will continue to investigate this issue. Also, will it be possible for you to share the complete indexer log?

Thanks,
Varun

indexer

sorry for the link, the file was to big to upload here.

restarted happened around 1920.

@dor272,

This issue was not seen before. Can you kindly share the files
c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\re_pm_id_time_15451661358099101112_0.index\data.fdb.11 and c:\Program Files\Couchbase\Server\var\lib\couchbase\data\@2i\re_pm_id_time_15451661358099101112_0.index\data.fdb.12 so that we can debug the issue further.

Thanks,
Varun

@varun.velamuri

I don’t have these files any longer (had to delete and recreate the indices in order to continue my work).

I can share it next time it happens, or if there is any other way I can help let me know.

Thanks.

@dor272,

We need those database files for analysing corruption. Please share the files when you are seeing the issue again.

Thanks,
Varun