An index creation got stuck

Couchbase Server
1

I am trying to create an index using Version: 4.5.0-2601 Community Edition (build-2601) , but it looks like it always got stuck.

The index which I am trying to create is:

CREATE INDEX ctx-index ON activity((distinct (array t for t within ctx end)))

The indexer log is full off errors:

[ns_server:error,2017-03-17T10:04:16.068Z,ns_1@127.0.0.1:index_stats_collector-index<0.1068.0>:index_rest:get_json:42]Request to (indexer) http://127.0.0.1:9102/stats?async=true failed: {error,
        {econnrefused,
        [{lhttpc_client,
          send_request,
          1,
          [{file,
            "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
           {line,
            220}]},
         {lhttpc_client,
          execute,
          9,
          [{file,
            "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
           {line,
            169}]},
         {lhttpc_client,
          request,
          9,
          [{file,
            "/home/couchbase/jenkins/workspace/watson-unix/couchdb/src/lhttpc/lhttpc_client.erl"},
           {line,
            92}]}]}}


2017-03-17T10:53:56.985+00:00 [Info] TK StreamBegin INIT_STREAM activity 247 160901367436594 1289
panic: runtime error: slice bounds out of range

goroutine 186 [running]:
panic(0xf293a0, 0xc820016050)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:464 +0x3e6 fp=0xc8234d31e0 sp=0xc8234d3160
runtime.panicslice()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/panic.go:21 +0x49 fp=0xc8234d3208 sp=0xc8234d31e0
github.com/couchbase/indexing/secondary/collatejson.(*Codec).code2json(0xc8234d3638, 0xc82376b1fa, 0x307b, 0x138, 0xc825a90000, 0x43ad, 0x7902, 0x0, 0x0, 0x0, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/collate.go:361 +0x159a fp=0xc8234d33d0 sp=0xc8234d3208
github.com/couchbase/indexing/secondary/collatejson.(*Codec).ExplodeArray(0xc8234d3638, 0xc823768201, 0x6074, 0x60ff, 0xc825a90000, 0x0, 0x7902, 0x0, 0x0, 0x0, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/collatejson/array.go:26 +0x27b fp=0xc8234d34c8 sp=0xc8234d33d0
github.com/couchbase/indexing/secondary/indexer.splitSecondaryArrayKey(0xc823768200, 0x6075, 0x6100, 0x0, 0xc825a90000, 0x0, 0x7902, 0x0, 0x0, 0x0, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/array.go:36 +0x19a fp=0xc8234d36a0 sp=0xc8234d34c8
github.com/couchbase/indexing/secondary/indexer.ArrayIndexItems(0xc823768200, 0x6075, 0x6100, 0x0, 0xc825a90000, 0x0, 0x7902, 0x1, 0x0, 0x0, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/array.go:78 +0xf3 fp=0xc8234d38e0 sp=0xc8234d36a0
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insertSecArrayIndex(0xc8220d5180, 0xc823768200, 0x6075, 0x6100, 0xc823762100, 0x606c, 0x6100, 0xc82366e780, 0x7, 0x7, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:513 +0xbf9 fp=0xc8234d3cd0 sp=0xc8234d38e0
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).insert(0xc8220d5180, 0xc823768200, 0x6075, 0x6100, 0xc823762100, 0x606c, 0x6100, 0xc82366e780, 0x7, 0x7, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:355 +0x14a fp=0xc8234d3d40 sp=0xc8234d3cd0
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).handleCommandsWorker(0xc8220d5180, 0x0)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:314 +0x4c6 fp=0xc8234d3f90 sp=0xc8234d3d40
runtime.goexit()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc8234d3f98 sp=0xc8234d3f90
created by github.com/couchbase/indexing/secondary/indexer.NewForestDBSlice
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:158 +0x1990

Not sure what it is trying to do with /home/couchbase/... path, this does not exists.

I also see some crash reports generated.

=erl_crash_dump:0.3
Fri Mar 17 09:07:31 2017
Slogan: Kernel pid terminated (application_controller) ({application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,net_kernel,{'EXIT',nodistribution}}}}},{k
System version: Erlang R16B03-1 (erts-5.10.4.0.0.1) [source-96430ba] [64-bit] [smp:4:4] [async-threads:16] [kernel-poll:false]
Compiled: Fri Mar 25 16:54:41 2016
Taints: 
Atoms: 4937
=memory
...
...

Any Ideas why it happened?

Also when I execute a query, I am getting following error:

[
  {
    "code": 5000,
    "msg": "dial tcp 127.0.0.1:9101: connection refused from 127.0.0.1:9101 - cause: dial tcp 127.0.0.1:9101: connection refused from 127.0.0.1:9101",
    "query_from_user": "SELECT count(*) as size FROM activity a WHERE  ANY t WITHIN `ctx` SATISFIES t LIKE 'Peter' END"
  }
]

There must be something wrong with an indexer engine. (I’ve tried restart)

indexer.log

goroutine 287 [chan receive]:
runtime.gopark(0x1322dc0, 0xc825895918, 0x10ce420, 0xc, 0x17, 0x3)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:262 +0x163 fp=0xc820040600 sp=0xc8200405d8
runtime.goparkunlock(0xc825895918, 0x10ce420, 0xc, 0x61e517, 0x3)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:268 +0x54 fp=0xc820040638 sp=0xc820040600
runtime.chanrecv(0xccd540, 0xc8258958c0, 0xc820040748, 0xc820040601, 0xc820040000)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/chan.go:470 +0x49f fp=0xc8200406c0 sp=0xc820040638
runtime.chanrecv2(0xccd540, 0xc8258958c0, 0xc820040748, 0xc820040701)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/chan.go:360 +0x2b fp=0xc8200406f0 sp=0xc8200406c0
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).waitPersist(0xc820136a00)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:1000 +0x20a fp=0xc8200407a0 sp=0xc8200406f0
github.com/couchbase/indexing/secondary/indexer.(*fdbSlice).IsDirty(0xc820136a00, 0x400)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/forestdb_slice_writer.go:1206 +0x21 fp=0xc8200407b0 sp=0xc8200407a0
github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker.func1(0xc8258b8cb0, 0xc8258def60, 0xc823cba460, 0xc8258def30, 0xc8201ab008, 0x8, 0xc820000001, 0x400, 0xc82499cc40, 0xc8258b8c9a, ...)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:311 +0x744 fp=0xc820040f40 sp=0xc8200407b0
runtime.goexit()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820040f48 sp=0xc820040f40
created by github.com/couchbase/indexing/secondary/indexer.(*storageMgr).createSnapshotWorker
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/indexing/secondary/indexer/storage_manager.go:396 +0x235

goroutine 339 [select]:
runtime.gopark(0x1322e00, 0xc82117a720, 0x10bbef8, 0x6, 0x18, 0x2)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/proc.go:262 +0x163 fp=0xc82117a478 sp=0xc82117a450
runtime.selectgoImpl(0xc82117a720, 0x0, 0x18)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/select.go:392 +0xa67 fp=0xc82117a638 sp=0xc82117a478
runtime.selectgo(0xc82117a720)
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/select.go:215 +0x12 fp=0xc82117a658 sp=0xc82117a638
github.com/couchbase/gometa/protocol.(*messageListener).start(0xc824390c30)
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:403 +0x5a3 fp=0xc82117a7b8 sp=0xc82117a658
runtime.goexit()
	/home/couchbase/.cbdepscache/exploded/x86_64/go-1.6/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc82117a7c0 sp=0xc82117a7b8
created by github.com/couchbase/gometa/protocol.(*Leader).AddWatcher
	/home/couchbase/jenkins/workspace/watson-unix/goproj/src/github.com/couchbase/gometa/protocol/leader.go:256 +0x5bc
[goport] 2017/03/17 15:00:46 /opt/couchbase/bin/indexer terminated: signal: aborted (core dumped)

Edit: it looks like it relates with
https://issues.couchbase.com/browse/MB-18930
I see the Fix version 4.5, but to me it seems to be not fixed.

2

@petojurkovic, It appears you are running into a crash due to sizing issue as reported in https://issues.couchbase.com/browse/MB-22081.

You can try changing the setting “max_array_seckey_size” to overcome the crash:

  1. Calculate the setting value:
    If the index definition is: create index idx on bucket( a, distinct (array (x.y) for x in b end) , c )
    then max_array_seckey_size = ( size ( a ) + size ( y ) + size ( c ) ) * (num of elements in b array)
    Here all sizes are in bytes. The default value is 10240 bytes (ie 10 KB).

  2. Apply the new setting:
    curl -X POST -u user:pass http://:9102/settings --data ‘{“indexer.settings.max_array_seckey_size”: size_in_bytes}’

1 Like