my indexes are not loading keep failing
here is the log result
Service ‘indexer’ exited with status 2. Restarting. Messages:
github.com/couchbase/indexing/secondary/common.CrashOnError(0x1532780, 0xc0457de310)
goproj/src/github.com/couchbase/indexing/secondary/common/util.go:406 +0x63
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleStorageWarmupDone(0xc0440a6000, 0x15336c0, 0xc046fc80a0)
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:4694 +0x9af
github.com/couchbase/indexing/secondary/indexer.(*indexer).handleWorkerMsgs(0xc0440a6000, 0x15336c0, 0xc046fc80a0)
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:950 +0x1938
github.com/couchbase/indexing/secondary/indexer.(*indexer).run(0xc0440a6000)
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:618 +0x182
github.com/couchbase/indexing/secondary/indexer.NewIndexer(0xc04210e3f0, 0x39, 0xc0000001ed, 0x0, 0x0)
goproj/src/github.com/couchbase/indexing/secondary/indexer/indexer.go:337 +0x1f33
main.main()
@muhammad_ibrahim Can you please share the panic stack trace from indexer.log located from the indexer node that crashed?
Hi from where i can find this i am sorry i dont know stack trace
Depends on the configured log location for the node. Typically it is /opt/couchbase/var/lib/couchbase/logs where you can find indexer.log. Please share indexer.log from above location from the indexer node which is failing.
Thanks,
Prathibha
HI the file is too much big but here is the top part
2020-04-01T22:29:14.701+05:00 [Info] Indexer::NewIndexer Build Mode Set Community
2020-04-01T22:29:14.701+05:00 [Info] Indexer::Cluster Storage Mode Set forestdb
2020-04-01T22:29:14.701+05:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
2020-04-01T22:29:14.712+05:00 [Info] SettingsMgr:: Dropped request /indexing/settings/config {“indexer.settings.compaction.days_of_week”:“Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday”,“indexer.settings.compaction.interval”:“00:00,00:00”,“indexer.settings.log_level”:“info”,“indexer.settings.persisted_snapshot.interval”:5000,“indexer.settings.compaction.compaction_mode”:“circular”,“indexer.settings.compaction.min_frag”:30,“indexer.settings.inmemory_snapshot.interval”:200,“indexer.settings.max_cpu_percent”:0,“indexer.settings.storage_mode”:“forestdb”,“indexer.settings.recovery.max_rollbacks”:5,“indexer.settings.memory_quota”:1048576000,“indexer.settings.compaction.abort_exceed_interval”:false}. Any setting change will get applied once Indexer is ready.
2020-04-01T22:29:14.717+05:00 [Info] [Queryport “:9101”] started …
2020-04-01T22:29:14.720+05:00 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap
2020-04-01T22:29:14.725+05:00 [Info] bootstrap storage mode forestdb
2020-04-01T22:29:14.773+05:00 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default
2020-04-01T22:29:14.775+05:00 [Info] serviceChangeNotifier: received ServiceChangeNotification
2020-04-01T22:29:14.778+05:00 [Info] serviceChangeNotifier: received PoolChangeNotification
2020-04-01T22:29:14.786+05:00 [INFO][FDB] Forestdb blockcache size 681574400 initialized in 13998 us
2020-04-01T22:29:14.787+05:00 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\MetadataStore
2020-04-01T22:29:17.023+05:00 [Info] Indexer::indexer version 3
2020-04-01T22:29:17.023+05:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2020-04-01T22:29:17.023+05:00 [Info] Indexer Id 3e73cf5ebae943f077f4d0594463c734
2020-04-01T22:29:17.052+05:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_3e73cf5ebae943f077f4d0594463c734
2020-04-01T22:29:17.053+05:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_3e73cf5ebae943f077f4d0594463c734. Err projector.topicMissing
2020-04-01T22:29:17.053+05:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2020-04-01T22:29:17.053+05:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2020-04-01T22:29:17.073+05:00 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_3e73cf5ebae943f077f4d0594463c734
2020-04-01T22:29:17.074+05:00 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_3e73cf5ebae943f077f4d0594463c734. Err projector.topicMissing
2020-04-01T22:29:17.074+05:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2020-04-01T22:29:17.074+05:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2020-04-01T22:29:17.074+05:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning
2020-04-01T22:29:17.074+05:00 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken
2020-04-01T22:29:17.074+05:00 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken
2020-04-01T22:29:17.074+05:00 [Info] ClustMgr:handleGetGlobalTopology &{map}
2020-04-01T22:29:17.075+05:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
Is possible, can you zip the file attach it here? Else, please look for word panic and paste the panic message and top part of the stack trace here. Thanks!
here the zip file of my indexer . right now couchbase is not letting me to even delete indexes so i can create it again . Thanks
Hi prathibha can you please tell me how to fix this issue … i am afraid if this happened in production. my cocuhbase server version is 6.0 community edition. Thanks
@muhammad_ibrahim: I saw the earlier attached indexer log file. I see that there are errors while reading from the storage index file of many indexes as part indexer process’s boot strap. Some of the errors that I see are:
2020-04-02T11:25:57.051+05:00 [ERRO][FDB] Crash Detected: Last Block not DBHEADER 0 in a database file ‘c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\stitchit_data_bucket_adv_type_date_branch_id_employee_id_11702943441739537100_0.index\data.fdb.0’^M
2020-04-02T11:26:03.370+05:00 [ERRO][FDB] Error in reading the doc length metadata with offset 2240917 from a database file ‘c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\stitchit_data_bucket_adv_type_register_id_15811407143511337398_0.index\data.fdb.1’^M
These errors while reading the index data files is causing the indexer to repeatedly crash , hence you’re unable to drop the indexes . Before suggesting a solution/workaround, would like to understand the issue the data files has run into. I am working with engineer from another team to debug this issue. Will get back with updates from debugging.
Thanks,
Prathibha
ok thanks i am waiting for the solution right now i am not be able to work in my dev server i am working on another couchbase server. i dont want to uninstall couchbase and install it again. i need the proper solution so incase in future this error occured in production server i can solve it y self . i cant unistall there hope you understand.
Thanks
Hi ,
Any solution for this issue . or can you please tell me how to delete indexes any possible way so i can recreate them. 
Hi, can you please provide a little more info about your setup?
- How many indexer nodes are there?
- How many indexes are there in the buckets?
- Are there any equivalent indexes created for each of the index? (Equivalent indexes are those that have same index definition but different index name)
One way to get out of this situation is :
- Stop couchbase server on the indexer node
- Delete index directories in /opt/couchbase/var/lob/couchbase/data/@2i path of all affected indexes. The index directory name has format bucketName_indexName_indexID_0.index. I see many indexes of different buckets running into errors. You can try deleting one index from each bucket. Note that, deleting an index file/dir of one index of a bucket will cause a rebuild of all indexes in that bucket (when indexer restarts). Also, be sure to delete only index directories and not other files in above path.
- Once you are done deleting, start the couchbase server.
After executing above steps, the indexes will get rebuilt.
Also, not being able to drop index when indexer has run into this of issue has been addressed in 6.5.0. In 6.5, you should be able to drop the indexes and restart the indexer which will take care of dropping the affected indexes. After which you need to recreate the indexes.
1 Like
Thanks deleted all the indexes from couchbase/var/lob/couchbase/data/@2i. but i hope this issue is fixed in 6.5 otherwise for me its gona be disaster in production for me 
i personally love couchbase
After starting couchbase server , are your indexes back up and built? We are yet to debug the actual root cause of loading up index files. Will get back on that one.
No after restarting the server all my indexes got deleted . i have created a script to generate the indexes . so i run the script and re created the indexes. i dont know this solution is good or not for production if such issue i faced again
but now every thing is fine thanks for support
This happened on my 2 machines . 1 is running with windows 10 and second one is running on windows server 2019. both using couchbase 6.0 . i sent you the logs of window 10 machine if you want me to send you the logs of windows server i can . but on my other 8 machines running same windows 10 and running couchbase 6.5 working fine. both couchbase 6.0 indexes were crashed together even there is not connection between both of them. i mean there they are running seperatly . one is running on my local laptop and windows server is one remote location . sorry both are on windows 10 pro