Unable to create index in couchbase

richa.gupta · July 26, 2018, 12:45pm

Steps executed:

Create a bucket Student and inserted json data.
Executed following query to create indexes but none of them is giving success result.
a. CREATE PRIMARY INDEX ON Student USING GSI WITH {“defer_build”:true}

Error:
[
{
“code”: 5000,
“msg”: “GSI CreatePrimaryIndex() - cause: Encounter errors during create index. Error=Terminate Request due to server termination\n.”,
“query_from_user”: “CREATE PRIMARY INDEX ON Student USING GSI WITH {“defer_build”:true}”
}
]

I do not understand what I am missing. I am badly stuck. Please assist asap.

vsr1 · July 26, 2018, 1:24pm

Make sure appropriate ports are opened. cc @deepkaran.salooja

deepkaran.salooja · July 26, 2018, 6:35pm

@richa.gupta, most likely one of the ports required by index service is already being used by some other process on the box. You can get the full list here.

You can check the indexer.log to find out the exact error and the port causing the conflict.

richa.gupta · July 26, 2018, 9:28pm

How can I check for open ports?

deepkaran.salooja · July 26, 2018, 10:41pm

I think it would be good to first check in the indexer.log about the exact error.

richa.gupta · July 27, 2018, 7:18am

I cannot attach the full indexer.log at the moment due to profile limitations. Below is the excerpt from indexer.log:

2018-07-27T12:51:59.159+05:30 [Info] Indexer started with command line: [c:/Program Files/Couchbase/Server/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=c:/Program Files/Couchbase/Server/var/lib/couchbase/data/@2i -diagDir=c:/Program Files/Couchbase/Server/var/lib/couchbase/crash -nodeUUID=99245c60f3d991a249becf11fee89a81 -isEnterprise=true -storageMode=plasma --httpsPort=19102 --certFile=c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached-cert.pem --keyFile=c:/Program Files/Couchbase/Server/var/lib/couchbase/config/memcached-key.pem]
2018-07-27T12:51:59.184+05:30 [Info] Indexer::Cluster Storage Mode Set plasma
2018-07-27T12:51:59.194+05:30 [Info] Indexer::NewIndexer Status Warmup
2018-07-27T12:51:59.417+05:30 [Info] Setting buffer block size to 16384 bytes
2018-07-27T12:51:59.417+05:30 [Info] Setting maxcpus = 4
2018-07-27T12:51:59.417+05:30 [Info] Setting log level to Info
2018-07-27T12:51:59.417+05:30 [Info] Indexer::NewIndexer Starting with Vbuckets 1024
2018-07-27T12:51:59.417+05:30 [Info] Indexer::NewIndexer Build Mode Set Enterprise
2018-07-27T12:51:59.426+05:30 [Info] [Queryport “:9101”] started …
2018-07-27T12:51:59.688+05:30 [Info] New settings received:
{“indexer.settings.compaction.days_of_week”:“Sunday,Monday,Tuesday,Wednesday,Thursday,Friday,Saturday”,“indexer.settings.compaction.interval”:“00:00,00:00”,“indexer.settings.compaction.compaction_mode”:“circular”,“indexer.settings.persisted_snapshot.interval”:5000,“indexer.settings.log_level”:“info”,“indexer.settings.compaction.min_frag”:30,“indexer.settings.inmemory_snapshot.interval”:200,“indexer.settings.max_cpu_percent”:0,“indexer.settings.storage_mode”:“plasma”,“indexer.settings.recovery.max_rollbacks”:5,“indexer.settings.memory_quota”:2040528896,“indexer.settings.compaction.abort_exceed_interval”:false}
2018-07-27T12:51:59.688+05:30 [Info] Setting maxcpus = 4
2018-07-27T12:51:59.688+05:30 [Info] Setting log level to Info
2018-07-27T12:51:59.689+05:30 [Info] Unable to fetch storage mode from metakv during bootrap. Use storage mode setting for bootstrap
2018-07-27T12:51:59.937+05:30 [Info] bootstrap storage mode plasma
2018-07-27T12:52:02.386+05:30 [INFO][FDB] Forestdb blockcache size 1048576 initialized in 0 us

2018-07-27T12:52:02.387+05:30 [INFO][FDB] Forestdb opened database file c:\Program Files\Couchbase\Server\var\lib\couchbase\data@2i\MetadataStore
2018-07-27T12:52:02.389+05:30 [Info] servicesChangeNotifier: Creating new notifier instance for http://%40index-cbauth@127.0.0.1:8091, default
2018-07-27T12:52:02.392+05:30 [Info] serviceChangeNotifier: received ServiceChangeNotification
2018-07-27T12:52:02.397+05:30 [Info] serviceChangeNotifier: received PoolChangeNotification
2018-07-27T12:52:04.676+05:30 [Info] Indexer::indexer version 2
2018-07-27T12:52:04.676+05:30 [Info] ClustMgr:handleGetLocalValue Key IndexerId
2018-07-27T12:52:04.676+05:30 [Info] Indexer Id 99245c60f3d991a249becf11fee89a81
2018-07-27T12:52:04.676+05:30 [Info] PeriodicStats = {“cpu_utilization”:0,“index_not_found_errcount”:0,“indexer_state”:“Warmup”,“memory_quota”:2040528896,“memory_used”:0,“memory_used_queue”:0,“memory_used_storage”:0,“needs_restart”:false,“num_connections”:0,“num_cpu_core”:4,“storage_mode”:“not_set”,“timings/stats_response”:“0 0 0”,“uptime”:“5.5168975s”}
2018-07-27T12:52:04.707+05:30 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_99245c60f3d991a249becf11fee89a81
2018-07-27T12:52:04.713+05:30 [Info] indexer:: SSL server started: :19102
2018-07-27T12:52:04.713+05:30 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic MAINT_STREAM_TOPIC_99245c60f3d991a249becf11fee89a81. Err projector.topicMissing
2018-07-27T12:52:04.713+05:30 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2018-07-27T12:52:04.713+05:30 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success
2018-07-27T12:52:04.732+05:30 [Info] KVSender::sendShutdownTopic Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_99245c60f3d991a249becf11fee89a81
2018-07-27T12:52:04.733+05:30 [Error] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 127.0.0.1:9999 Topic INIT_STREAM_TOPIC_99245c60f3d991a249becf11fee89a81. Err projector.topicMissing
2018-07-27T12:52:04.733+05:30 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 127.0.0.1:9999
2018-07-27T12:52:04.733+05:30 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success
2018-07-27T12:52:04.733+05:30 [Info] ClustMgr:handleGetLocalValue Key RebalanceRunning
2018-07-27T12:52:04.733+05:30 [Info] ClustMgr:handleGetLocalValue Key RebalanceToken
2018-07-27T12:52:04.733+05:30 [Info] Indexer::recoverRebalanceState RebalanceRunning false RebalanceToken
2018-07-27T12:52:04.733+05:30 [Info] ClustMgr:handleGetGlobalTopology &{map[]}
2018-07-27T12:52:04.734+05:30 [Info] Indexer::initFromPersistedState Recovered IndexInstMap
InstanceId: 16033610815315464632 Name: def_type Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 15081929406794695465 Name: def_sourceairport Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 13412169846185038080 Name: def_faa Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 6188326768972702402 Name: def_city Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 13613033200967328094 Name: def_icao Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 10177609192700347780 Name: def_schedule_utc Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 787542517909168144 Name: def_airportname Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 10529725552457111256 Name: def_name_type Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 14744973601169109270 Name: def_primary Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
InstanceId: 812528154559171700 Name: def_route_src_dst_day Bucket: travel-sample State: INDEX_STATE_ACTIVE Stream: MAINT_STREAM RState: RebalActive Version: 0 ReplicaId: 0
2018-07-27T12:52:04.780+05:30 [Info] {bucket,feeds} “travel-sample” created for dcp_seqno cache…
2018-07-27T12:52:04.781+05:30 [Info] Indexer::getCurrentKVTs Time Taken 23.0165ms
2018-07-27T12:52:04.782+05:30 [Info] indexer.upgradeStorage: check index for storage upgrade. disable false overrride not_set
2018-07-27T12:52:04.782+05:30 [Info] Indexer::local storage mode plasma
2018-07-27T12:52:04.782+05:30 [Info] Indexer::initPartnInstance Initialized Partition:
Index: 15081929406794695465 Partition: PartitionId: 0 Endpoints: [:9105]
panic: runtime error: index out of range

Please assist

deepkaran.salooja · July 27, 2018, 4:41pm

@richa.gupta, can you also share the stack just after the panic. Also which couchbase version are you using?

You can upload the indexer.log here:

curl -X PUT -T indexer.log https://forumlogs.s3-us-west-1.amazonaws.com/indexer.log

richa.gupta · July 30, 2018, 4:45am

indexer.zip (90.2 KB)

Attached indexer.log.
Couchbase Server 5.1

deepkaran.salooja · July 30, 2018, 6:26am

You seem to have run into a known issue where index disk file can get corrupted if the machine powers off suddenly. We have seen this issue mostly on laptops. You can read more details here:

https://issues.couchbase.com/browse/MB-25086?focusedCommentId=222525&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-222525

You’ll need to reinstall couchbase on the machine.

richa.gupta · July 31, 2018, 7:01am

@deepkaran.salooja
It really helped. I can create indexes now after reinstallation. Thanks a lotttt