[info] source-based builds: 4.0.0 - ok, 4.1.0 - ok, 4.1.1 - indexer performance is terribly slow

POST-EDITION: source based-builds of 4.0.0 and 4.1.0 (released ones) both have no problem with indexer, so i suppose, my problem, described below, most likely because 4.1.1 (manifests/couchbase-server/sherlock/4.1.1.xml) is really a “developing set of commits” :slight_smile: Looking forward to test 4.1.1 after release. And special thanks to all “tlm” contributors: trondn, daverigby, ceejatec and others; you made a really great thing!

ORIGINAL CONTENTS:
Disclaimer: of course, i understand that 4.1.1 is not officially released (i.e. this could be just a “untested set of commits”), but the manifest is available on github, so “why not ?”. My 4.1.1-build, based of this manifest, runs fine and produces no “explicit” errors (built with RelWithDebugInfo).

Comparing 4.1.1 source-based build (repo state before build is described below; it’s according to used manifest) with 4.0.0-4051 CE on 3-node cluster (same application code is used). For design docs on both clusters buckets {“updateInterval”:1000,“updateMinChanges”:1,“replicaUpdateMinChanges”:1} are set (“updateMinChanges”:1,“replicaUpdateMinChanges”:1 => for used design docs, “updateInterval”:1000 for viewUpdateDaemon; this is globally, as i understand)

Problem: 4.0.0 CE works normal while indexing (less then 10 ops/sec); 4.1.1 is just unable to index fast enough => application tests show errors; output shows that view call returning not all results it should (comparing to same tests for CE). Even “sitting and watching ‘views’ web-interface page”, i see “indexing …%” is slooooow on 4.1.1; and “almost don’t see ‘indexing %’ on CE” during applications tests.

Please, help me to understand:

  1. what external libraries/dependencies has impact on indexer performance ?
  2. is there any “reference build environment” to build couchbase from source ?
  3. what can possibly help to increase indexer performance ?

repo info (before build):
Manifest branch: master
Manifest merge branch: refs/heads/master
Manifest groups: default,platform-linux

Project: couchbase-cli
Mount path: /root/couchbase/src/couchbase-cli
Current revision: 32cc813064ef83e73c212a48b589be035bc6265c
Local Branches: 0

Project: couchbase-examples
Mount path: /root/couchbase/src/couchbase-examples
Current revision: 2e9cd32988929935c9b172fb713434c1f81243de
Local Branches: 0

Project: couchbase-python-client
Mount path: /root/couchbase/src/couchbase-python-client
Current revision: d3e69f6c1eb9ec8fbc240b79bb228fe366e8295f
Local Branches: 0

Project: couchdb
Mount path: /root/couchbase/src/couchdb
Current revision: d5111e5907523633e8e56c615e09f0f58940c6cb
Local Branches: 0

Project: couchstore
Mount path: /root/couchbase/src/couchstore
Current revision: 7d6bc22a15b80f1da6dd169a3e538e985db1b91a
Local Branches: 0

Project: ep-engine
Mount path: /root/couchbase/src/ep-engine
Current revision: 00272e095b58fb5267b950b0511a0b9147b40dd5
Local Branches: 0

Project: forestdb
Mount path: /root/couchbase/src/forestdb
Current revision: 1ea5fd5bfe3a38b346be4f23f507ea4ea8a27733
Local Branches: 0

Project: geocouch
Mount path: /root/couchbase/src/geocouch
Current revision: a8b9540e38ca9c116fadfe7c3d163f693b3b2300
Local Branches: 0

Project: goprotobuf
Mount path: /root/couchbase/src/godeps/src/code.google.com/p/goprotobuf
Current revision: 682707530d5a3d6cb757dfe8fa322d0fde7621c6
Local Branches: 0

Project: cbauth
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/cbauth
Current revision: 1323b92ac2619c29d50e588e59d7a6b4839da629
Local Branches: 0

Project: clog
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/clog
Current revision: e0f356188f6f2f3098eb7dbc9082801dffc3f1a8
Local Branches: 0

Project: dparval
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/dparval
Current revision: 9def03782da875a2477c05bf64985db3f19f59ae
Local Branches: 0

Project: go-couchbase
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/go-couchbase
Current revision: 162c266a6106641f182685b9b8e618a871936269
Local Branches: 0

Project: go-slab
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/go-slab
Current revision: 1f5f7f282713ccfab3f46b1610cb8da34bcf676f
Local Branches: 0

Project: goforestdb
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/goforestdb
Current revision: 5e58436197098dfc996ed028afb6005195cf3450
Local Branches: 0

Project: gomemcached
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/gomemcached
Current revision: 182dfcc4f567a714c57a954b942669707c2f129a
Local Branches: 0

Project: retriever
Mount path: /root/couchbase/src/godeps/src/github.com/couchbase/retriever
Current revision: e3419088e4d3b4fe3aad3b364fdbe9a154f85f17
Local Branches: 0

Project: go-jsonpointer
Mount path: /root/couchbase/src/godeps/src/github.com/dustin/go-jsonpointer
Current revision: 75939f54b39e7dafae879e61f65438dadc5f288c
Local Branches: 0

Project: gojson
Mount path: /root/couchbase/src/godeps/src/github.com/dustin/gojson
Current revision: af16e0e771e2ed110f2785564ae33931de8829e4
Local Branches: 0

Project: protobuf
Mount path: /root/couchbase/src/godeps/src/github.com/golang/protobuf
Current revision: 655cdfa588ea190e901bc5590e65d5621688847c
Local Branches: 0

Project: context
Mount path: /root/couchbase/src/godeps/src/github.com/gorilla/context
Current revision: 215affda49addc4c8ef7e2534915df2c8c35c6cd
Local Branches: 0

Project: mux
Mount path: /root/couchbase/src/godeps/src/github.com/gorilla/mux
Current revision: 8a875a034c69b940914d83ea03d3f1299b4d094b
Local Branches: 0

Project: npipe
Mount path: /root/couchbase/src/godeps/src/github.com/natefinch/npipe
Current revision: 272c8150302e83f23d32a355364578c9c13ab20f
Local Branches: 0

Project: go-metrics
Mount path: /root/couchbase/src/godeps/src/github.com/rcrowley/go-metrics
Current revision: dee209f2455f101a5e4e593dea94872d2c62d85d
Local Branches: 0

Project: go-zookeeper
Mount path: /root/couchbase/src/godeps/src/github.com/samuel/go-zookeeper
Current revision: fa6674abf3f4580b946a01bf7a1ce4ba8766205b
Local Branches: 0

Project: liner
Mount path: /root/couchbase/src/godeps/src/github.com/sbinet/liner
Current revision: d9335eee40a45a4f5d74524c90040d6fe6013d50
Local Branches: 0

Project: gometa
Mount path: /root/couchbase/src/goproj/src/github.com/couchbase/gometa
Current revision: 7451ab4fa283b755eb083b4cc2fcf824b25e0756
Local Branches: 0

Project: goxdcr
Mount path: /root/couchbase/src/goproj/src/github.com/couchbase/goxdcr
Current revision: 45cbb7086fe6f22400e5440ad66501a604ab1b6a
Local Branches: 0

Project: indexing
Mount path: /root/couchbase/src/goproj/src/github.com/couchbase/indexing
Current revision: d9fdc07474d0def446fdf2c3204d57e59718c079
Local Branches: 0

Project: query
Mount path: /root/couchbase/src/goproj/src/github.com/couchbase/query
Current revision: 64b2cc88f4d0b1b7e0a0dea39c5e4e1b2d52c67a
Local Branches: 0

Project: memcached
Mount path: /root/couchbase/src/memcached
Current revision: 0c77a90723d6ab5e2991baac720b393c87b99c04
Local Branches: 0

Project: moxi
Mount path: /root/couchbase/src/moxi
Current revision: 5ab60b49048b8f6d93386a30ff743a8e5daa7c7a
Local Branches: 0

Project: ns_server
Mount path: /root/couchbase/src/ns_server
Current revision: 46a4fdd2979cfe7820787047d6ac4baed3222d3a
Local Branches: 0

Project: platform
Mount path: /root/couchbase/src/platform
Current revision: d91fece94c69331e876a8d18b2d75090fbf88848
Local Branches: 0

Project: sigar
Mount path: /root/couchbase/src/sigar
Current revision: 83bd88595f558987731a1baa3330f641bc36554c
Local Branches: 0

Project: testrunner
Mount path: /root/couchbase/src/testrunner
Current revision: 325345ea7245121c791eb0369fb79eafb09fa657
Local Branches: 0

Project: tlm
Mount path: /root/couchbase/src/tlm
Current revision: 24d44145c4edbf540bc65a4b0122c6c2abf6d38e
Local Branches: 0