Couchbase Server Community 4.1.1
Sync Gateway Community 1.3.1
Sync gateway crashed for us recently and it caused a crash of our couchbase node as well with. Right before the error in the sync gateway logs I saw it write to a doc with 2820 revisons. We have no max revisions set, so the default max of 1000 should be applied, but in the _sync object in the doc I can see it has all revisions listed in history and records of conflicts in bodymap for about 4500 revision conflicts. Could writing to this huge doc have caused a crash? Is there anyway to force sync gateway to discard old revision info in the _sync object so it only stores about a 100 revisions?
This log message was in the couchbase server logs:
Service ‘memcached’ exited with status 137. Restarting. Messages: 2019-09-09T17:14:01.461912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) stream created!
2019-09-09T17:14:01.462115Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 121415 was last seqno sent, reason: The stream ended due to all items being streamed
2019-09-09T17:14:01.463876Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Creating stream with start seqno 129890 and end seqno 129891
2019-09-09T17:14:01.463912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) stream created!
2019-09-09T17:14:01.464091Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 129891 was last seqno sent, reason: The stream ended due to all items being streamed
Sync gateway log error message:
http: panic serving 127.0.0.1:37224: runtime error: invalid memory address or nil pointer dereference
goroutine 41825347 [running]:
net/http.(*conn).serve.func1(0xc8a164a9a0, 0x7fd4bc33ef30, 0xc868698b38)
/usr/local/go/src/net/http/server.go:1287 +0xb5
github.com/couchbase/gomemcached/client.(*Client).CASNext(0xc828d5b290, 0xc8b78301b4, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0xc8b78387f0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/gomemcached/client/mc.go:595 +0x15e
github.com/couchbase/go-couchbase.(*Bucket).casNext.func2(0xc828d5b290, 0xf801b4, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:966 +0x7b
github.com/couchbase/go-couchbase.(*Bucket).Do.func1(0xc82039a700, 0xc8000001b4, 0xc8bc1a4cb8, 0xc8202ac500, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:113 +0x127
github.com/couchbase/go-couchbase.(*Bucket).Do(0xc82039a700, 0xc8521abde0, 0x17, 0xc8bc1a4cb8, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:119 +0x27c
github.com/couchbase/go-couchbase.(*Bucket).casNext(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0x89bd00)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:968 +0x12b
github.com/couchbase/go-couchbase.(*Bucket).update(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4df0, 0xf7eea1, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:997 +0x85
github.com/couchbase/go-couchbase.(*Bucket).WriteUpdate(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4e50, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:1025 +0xba
github.com/couchbase/sync_gateway/base.CouchbaseBucket.WriteUpdate(0xc82039a700, 0xc820191060, 0x15, 0xd45280, 0x7, 0xc82026a770, 0x5, 0x0, 0x0, 0x0, …)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/base/bucket.go:118 +0x8a
github.com/couchbase/sync_gateway/base.(*CouchbaseBucket).WriteUpdate(0xc82006a4d0, 0xc8521abde0, 0x17, 0x0, 0xc8bbf69830, 0x0, 0x0)
:100 +0xf3
github.com/couchbase/sync_gateway/db.(*Database).updateDoc(0xc8521ab7c0, 0xc8521abde0, 0x17, 0x0, 0xc82e967680, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:535 +0x54b
github.com/couchbase/sync_gateway/db.(*Database).PutExistingRev(0xc8521ab7c0, 0xc8521abde0, 0x17, 0xc8b189db30, 0xc8a2a2b480, 0x2, 0x2, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:513 +0x466
github.com/couchbase/sync_gateway/rest.(*handler).handleBulkDocs(0xc8866f58c0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/bulk_api.go:422 +0x1dc8
github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc8866f58c0, 0xf82f08, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:200 +0x7b4
github.com/couchbase/sync_gateway/rest.makeHandler.func1(0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:90 +0x77
net/http.HandlerFunc.ServeHTTP(0xc82036ed00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc8205ad950, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/gorilla/mux/mux.go:86 +0x2a6
github.com/couchbase/sync_gateway/rest.wrapRouter.func1(0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/routing.go:299 +0x32d
net/http.HandlerFunc.ServeHTTP(0xc820453140, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc82056fb00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8a164a9a0)
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6