4.1.1[go crash] vs 4.1.2[just error]: create primary index error

[UPDATE]

  • 4.1.1 src-based build has go GO crash on “create primary index” (see this post, or first part of second long reply)
  • 4.1.2 src-based build has error “[Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector” (second part of second long reply)
  • 4.1.1-EE is ok, no errors.

Has anyone tried to build 4.1.1 (or probably 4.1.2) and make indexes ?
Or am i really “the only one strange dude who tries to make source-based builds” ?

P.S. Probably, it’s a good idea to make a poll and ask people “what build (not version, but build: CE,EE, source-based) of couchbase do you use ?” …

cbq> create primary index on bucket.name;
{
“requestID”: “4a25c741-6764-4fa5-9439-35b56ac48832”,
“signature”: null,
“results”: [
],
“errors”: [
{
“code”: 5000,
“msg”: “GSI CreatePrimaryIndex() - cause: Terminate Request due to server termination”
}
],
“status”: “errors”,
“metrics”: {
“elapsedTime”: “222.206682ms”,
“executionTime”: “221.719686ms”,
“resultCount”: 0,
“resultSize”: 0,
“errorCount”: 1
}
}

Indexer.log

2016-05-03T14:39:01.54+00:00 [Error] panic in Leader.listen() : runtime error: cgo argument has Go pointer to Go pointer
2016-05-03T14:39:01.54+00:00 [Error] /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24
github.com/couchbase/gometa/protocol.(*Leader).listen.func1(0xc82015a510)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:456 +0x10f
panic(0xe0b320, 0xc8240a7260)
/usr/local/go/src/runtime/panic.go:443 +0x4e9
github.com/couchbase/indexing/secondary/fdb._cgoCheckPointer0(0xc09b80, 0xc8244a0a74, 0x0, 0x0, 0x0, 0xc8240afa60)
??:0 +0x4d
github.com/couchbase/indexing/secondary/fdb.(*KVStore).SetKV(0xc820159410, 0xc8240afa60, 0x13, 0x20, 0xc8244a0a74, 0x31, 0x40, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/indexing/secondary/fdb/kv.go:65 +0x1ba
github.com/couchbase/gometa/repository.(*Repository).Set(0xc822f80f00, 0x0, 0x100c360, 0x13, 0xc8244a0a74, 0x31, 0x40, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/repository/repo.go:136 +0x372
github.com/couchbase/gometa/action.(*ServerAction).persistChange(0xc82015cd90, 0x100000002, 0x100c360, 0x13, 0xc8244a0a74, 0x31, 0x40, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/action/action.go:373 +0xf1
github.com/couchbase/gometa/action.(*ServerAction).Commit(0xc82015cd90, 0x100000002, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/action/action.go:163 +0x192
github.com/couchbase/gometa/protocol.(*Leader).commit(0xc82015a510, 0x100000002, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:827 +0x6e5
github.com/couchbase/gometa/protocol.(*Leader).newProposal(0xc82015a510, 0x7f62105f44e0, 0xc8242983c0, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:596 +0x360
github.com/couchbase/gometa/protocol.(*Leader).createProposal(0xc82015a510, 0xc820155280, 0x5, 0x7f6210675dd0, 0xc82385c3c0, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:554 +0x3c5
github.com/couchbase/gometa/protocol.(*Leader).handleMessage(0xc82015a510, 0x7f6210665cb8, 0xc82385c3c0, 0xc820155280, 0x5, 0x0, 0x0)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:516 +0x29c
github.com/couchbase/gometa/protocol.(*Leader).listen(0xc82015a510)
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:475 +0x150
created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler
/root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:160 +0x3ad

4.1.1, source-based build. Same command for 4.1.1-EE works well.
Tried default Ubuntu 14.04 go(1.6.0) and 1.6.2 => same results.
Any ideas ?

Hi,

The correct syntax is:

CREATE PRIMARY INDEX ON my_bucket;

@geraldss (cool! now i know how to make name-references!),
it’s not a syntax problem. same command with EE works fine, no problems at all.

4.1.1, src-based build:
1: initialized 3-nodes cluster (both “index” and “query” services are enabled for each node)
2: created bucket “mybucket” (1 replica, view index replicas, full ejection, no password, disk io = high, flush = enable)
3: checked “index” menu item: “there are no indexes currently”, no messages.
4: opened consle, run cbq on first node:
cbq> CREATE PRIMARY INDEX ON mybucket; { "requestID": "af5bb77b-4581-4d0b-ab5a-0c550ea66f7c", "signature": null, "results": [ ], "errors": [ { "code": 5000, "msg": "GSI CreatePrimaryIndex() - cause: Terminate Request due to server termination" } ], "status": "errors", "metrics": { "elapsedTime": "214.55524ms", "executionTime": "214.496831ms", "resultCount": 0, "resultSize": 0, "errorCount": 1 } }
5: Looked at indexer.log:

2016-05-03T19:05:17.504+00:00 [Error] panic in Leader.listen() : runtime error: cgo argument has Go pointer to Go pointer 2016-05-03T19:05:17.504+00:00 [Error] /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/indexing/secondary/logging/logging.go:165 +0x24 github.com/couchbase/gometa/protocol.(*Leader).listen.func1(0xc82009c480) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:456 +0x10f panic(0xe0b320, 0xc823513d50) /usr/local/go/src/runtime/panic.go:443 +0x4e9 github.com/couchbase/indexing/secondary/fdb._cgoCheckPointer0(0xc09b80, 0xc822fe6ff4, 0x0, 0x0, 0x0, 0xc8231d1520) ??:0 +0x4d github.com/couchbase/indexing/secondary/fdb.(*KVStore).SetKV(0xc822fa0360, 0xc8231d1520, 0x13, 0x20, 0xc822fe6ff4, 0x2b, 0x40, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/indexing/secondary/fdb/kv.go:65 +0x1ba github.com/couchbase/gometa/repository.(*Repository).Set(0xc820010900, 0x0, 0x100c360, 0x13, 0xc822fe6ff4, 0x2b, 0x40, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/repository/repo.go:136 +0x372 github.com/couchbase/gometa/action.(*ServerAction).persistChange(0xc822f9e380, 0x100000002, 0x100c360, 0x13, 0xc822fe6ff4, 0x2b, 0x40, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/action/action.go:373 +0xf1 github.com/couchbase/gometa/action.(*ServerAction).Commit(0xc822f9e380, 0x100000002, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/action/action.go:163 +0x192 github.com/couchbase/gometa/protocol.(*Leader).commit(0xc82009c480, 0x100000002, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:827 +0x6e5 github.com/couchbase/gometa/protocol.(*Leader).newProposal(0xc82009c480, 0x7f73fc0067b8, 0xc8233a0cc0, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:596 +0x360 github.com/couchbase/gometa/protocol.(*Leader).createProposal(0xc82009c480, 0xc822f84570, 0x5, 0x7f73fbffcb60, 0xc822fd5040, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:554 +0x3c5 github.com/couchbase/gometa/protocol.(*Leader).handleMessage(0xc82009c480, 0x7f73fc03bc90, 0xc822fd5040, 0xc822f84570, 0x5, 0x0, 0x0) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:516 +0x29c github.com/couchbase/gometa/protocol.(*Leader).listen(0xc82009c480) /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:475 +0x150 created by github.com/couchbase/gometa/protocol.NewLeaderWithCustomHandler /root/couchbase-4.1.1/src/goproj/src/github.com/couchbase/gometa/protocol/leader.go:160 +0x3ad 2016-05-03T19:05:51.816+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":172032,"needs_restart":false,"num_connections":0,"timings/stats_response":"723 39288765 2465882730503"}

6: checked “index” menu item again: “Warning: We are having troubles communicating to the indexer process. The information might be stale” message exists.

Has anyone tried to build 4.1.1 and make indexes ?
Or am i really “the only one strange dude who tries to make source-based builds” ?

4.1.2, src-based build (dev-team):
1: initialized 3-nodes cluster (both “index” and “query” services are enabled for each node)
2: created bucket “mybucket” (1 replica, view index replicas, full ejection, no password, disk io = high, flush = enable)
3: checked “index” menu item: “there are no indexes currently”, no messages.
4: opened consle, run cbq on first node:
cbq> CREATE PRIMARY INDEX ON mybucket; { "requestID": "b3304ba2-c125-487f-a5a7-1243aa297164", "signature": null, "results": [ ], "errors": [ { "code": 5000, "msg": "GSI CreatePrimaryIndex() - cause: Terminate Request due to server termination" } ], "status": "errors", "metrics": { "elapsedTime": "203.691505ms", "executionTime": "203.167114ms", "resultCount": 0, "resultSize": 0, "errorCount": 1 } }
5: Looked at indexer.log (**WOW, no GO-crash anymore! **):
2016-05-03T19:38:25.636+00:00 [Info] Indexer started with command line: [/opt/couchbase/bin/indexer -vbuckets=1024 -cluster=127.0.0.1:8091 -adminPort=9100 -scanPort=9101 -httpPort=9102 -streamInitPort=9103 -streamCatchupPort=9104 -streamMaintPort=9105 -storageDir=/opt/couchbase/var/lib/couchbase/data/@2i -diagDir=/opt/couchbase/var/lib/couchbase/crash] 2016-05-03T19:38:25.728+00:00 [Info] Indexer::NewIndexer Status INIT 2016-05-03T19:38:25.74+00:00 [Info] Setting maxcpus = 4 2016-05-03T19:38:25.74+00:00 [Info] Setting buffer block size to 16384 bytes 2016-05-03T19:38:25.74+00:00 [Info] Setting log level to Info 2016-05-03T19:38:25.742+00:00 [Info] Indexer::NewIndexer Starting with Vbuckets 1024 2016-05-03T19:38:25.803+00:00 [Info] [Queryport ":9101"] started ... 2016-05-03T19:38:25.847+00:00 [Info] New settings received: {"indexer.settings.compaction.interval":"00:00,00:00","indexer.settings.persisted_snapshot.interval":5000,"indexer.settings.log_level":"info","indexer.settings.compaction.min_frag":30,"indexer.settings.inmemory_snapshot.interval":200,"indexer.settings.max_cpu_percent":400,"indexer.settings.recovery.max_rollbacks":5,"indexer.settings.memory_quota":268435456} 2016-05-03T19:38:25.847+00:00 [Info] Setting maxcpus = 4 2016-05-03T19:38:25.847+00:00 [Info] Setting log level to Info 2016-05-03T19:38:26.147+00:00 [Info] ClustMgr:handleGetLocalValue Key IndexerId 2016-05-03T19:38:26.148+00:00 [Info] ClustMgr:handleSetLocalValue Key IndexerId Value 6b:f5:e1:b0:66:99:15:db 2016-05-03T19:38:26.18+00:00 [Info] Indexer Id 6b:f5:e1:b0:66:99:15:db 2016-05-03T19:38:26.208+00:00 [Info] KVSender::sendShutdownTopic Projector 80.node:9999 Topic MAINT_STREAM_TOPIC_6b:f5:e1:b0:66:99:15:db 2016-05-03T19:38:26.21+00:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 80.node:9999 Topic MAINT_STREAM_TOPIC_6b:f5:e1:b0:66:99:15:db. Err projector.topicMissing 2016-05-03T19:38:26.21+00:00 [Error] KVSender::closeMutationStream MAINT_STREAM Error Received projector.topicMissing from 80.node:9999 2016-05-03T19:38:26.21+00:00 [Info] KVSender::closeMutationStream MAINT_STREAM Treating projector.topicMissing As Success 2016-05-03T19:38:26.219+00:00 [Info] KVSender::sendShutdownTopic Projector 80.node:9999 Topic INIT_STREAM_TOPIC_6b:f5:e1:b0:66:99:15:db 2016-05-03T19:38:26.219+00:00 [Fatal] KVSender::sendShutdownTopic Unexpected Error During Shutdown Projector 80.node:9999 Topic INIT_STREAM_TOPIC_6b:f5:e1:b0:66:99:15:db. Err projector.topicMissing 2016-05-03T19:38:26.219+00:00 [Error] KVSender::closeMutationStream INIT_STREAM Error Received projector.topicMissing from 80.node:9999 2016-05-03T19:38:26.219+00:00 [Info] KVSender::closeMutationStream INIT_STREAM Treating projector.topicMissing As Success 2016-05-03T19:38:26.219+00:00 [Info] ClustMgr:handleGetGlobalTopology &{map[]} 2016-05-03T19:38:26.219+00:00 [Info] Indexer::initFromPersistedState Recovered IndexInstMap 2016-05-03T19:38:26.22+00:00 [Info] MutationMgr::handleUpdateIndexInstMap Message: MsgUpdateInstMap 2016-05-03T19:38:26.22+00:00 [Info] MutationMgr::handleUpdateIndexPartnMap Message: MsgUpdatePartnMap 2016-05-03T19:38:26.22+00:00 [Info] ClustMgr:handleIndexerReady &{52 [] {false false false false} 0} 2016-05-03T19:38:26.22+00:00 [Info] Indexer::NewIndexer Status ACTIVE 2016-05-03T19:38:26.22+00:00 [Info] CompactionManager: Refreshing settings 2016-05-03T19:39:26.147+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"72 4455283 352661974371"} 2016-05-03T19:40:26.148+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"155 8582139 583045485273"} 2016-05-03T19:41:26.148+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"251 12822115 788758474501"} 2016-05-03T19:42:26.148+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"345 17745621 1124692913311"} 2016-05-03T19:43:26.149+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"417 21796571 1376779253647"} 2016-05-03T19:44:26.149+00:00 [Info] PeriodicStats = {"memory_quota":268435456,"memory_used":94208,"needs_restart":false,"num_connections":0,"timings/stats_response":"489 25865900 1633728523068"}

6: checked “index” menu item again: "Warning: We are having troubles communicating to the indexer process. The information might be stale. " message exists.

Hi @grep, which manifest are you using for 4.1.2 based build?
thanks

@cihangirb
This one: https://github.com/couchbase/build-team-manifests/blob/master/sherlock.xml

The answer is go 1.4.x for 4.1.X builds

I would recommend using these instead. they are more stable.

@cihangirb,
yep, thanks, i know about them.