The indexer service keeps crashing (core dumped) and restarting on a single server in our cluster.
I’d like to remove the indexes that are created on this one and re-create them elsewhere, but that fails too ("DROP INDEX..." fails with “GSI Drop() - cause: Terminate Request due to server termination”).
In the GUI it shows as “Bootstrap” all time time for all the indexes that are on this particular server.
In the indexer.log, it complains about not being able to open an index DB file: 2017-03-21T22:51:34.503-04:00 [Error] ForestDBSnapshot::Open Unexpected Error Opening Main DB Snapshot (/couchdata/data/@2i/[INDEX-NAME-REMOVED]_2235194985276477961_0.index) SeqNum 16015538 FDB_RESULT_NO_DB_INSTANCE panic: Unable to open snapshot -FDB_RESULT_NO_DB_INSTANCE
@uris, Looks like you have run into a storage issue unfortunately. You can stop couchbase service on the server, remove the contents of /couchdata/data/@2i and start the service again. This will force cleanup all the indexes on this server. and you can recreate them.
If you want the system to rebuild all the indexes on this server automatically, just leave the file MetadataStore in /couchdata/data/@2i and remove everything else.
btw (hijacking my own thread): is there a way to systematically get all the index definitions? I know I can run SELECT * FROM system:indexes but that doesn’t return where the indexes are created (it says 127.0.0.1 for all). Is there a more complete way to get the full definition of indexes so they can be backed-up and re-created if needed?