Can too many revisions on a document cause sync gateway or couchbase server to crash?

Couchbase Server Community 4.1.1
Sync Gateway Community 1.3.1

Sync gateway crashed for us recently and it caused a crash of our couchbase node as well with. Right before the error in the sync gateway logs I saw it write to a doc with 2820 revisons. We have no max revisions set, so the default max of 1000 should be applied, but in the _sync object in the doc I can see it has all revisions listed in history and records of conflicts in bodymap for about 4500 revision conflicts. Could writing to this huge doc have caused a crash? Is there anyway to force sync gateway to discard old revision info in the _sync object so it only stores about a 100 revisions?

This log message was in the couchbase server logs:
Service ‘memcached’ exited with status 137. Restarting. Messages: 2019-09-09T17:14:01.461912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) stream created!
2019-09-09T17:14:01.462115Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 121415 was last seqno sent, reason: The stream ended due to all items being streamed
2019-09-09T17:14:01.463876Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Creating stream with start seqno 129890 and end seqno 129891
2019-09-09T17:14:01.463912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) stream created!
2019-09-09T17:14:01.464091Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 129891 was last seqno sent, reason: The stream ended due to all items being streamed

Sync gateway log error message:
http: panic serving 127.0.0.1:37224: runtime error: invalid memory address or nil pointer dereference
goroutine 41825347 [running]:
net/http.(*conn).serve.func1(0xc8a164a9a0, 0x7fd4bc33ef30, 0xc868698b38)
/usr/local/go/src/net/http/server.go:1287 +0xb5
github.com/couchbase/gomemcached/client.(*Client).CASNext(0xc828d5b290, 0xc8b78301b4, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0xc8b78387f0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/gomemcached/client/mc.go:595 +0x15e
github.com/couchbase/go-couchbase.(*Bucket).casNext.func2(0xc828d5b290, 0xf801b4, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:966 +0x7b
github.com/couchbase/go-couchbase.(*Bucket).Do.func1(0xc82039a700, 0xc8000001b4, 0xc8bc1a4cb8, 0xc8202ac500, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:113 +0x127
github.com/couchbase/go-couchbase.(*Bucket).Do(0xc82039a700, 0xc8521abde0, 0x17, 0xc8bc1a4cb8, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:119 +0x27c
github.com/couchbase/go-couchbase.(*Bucket).casNext(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0x89bd00)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:968 +0x12b
github.com/couchbase/go-couchbase.(*Bucket).update(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4df0, 0xf7eea1, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:997 +0x85
github.com/couchbase/go-couchbase.(*Bucket).WriteUpdate(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4e50, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/go-couchbase/client.go:1025 +0xba
github.com/couchbase/sync_gateway/base.CouchbaseBucket.WriteUpdate(0xc82039a700, 0xc820191060, 0x15, 0xd45280, 0x7, 0xc82026a770, 0x5, 0x0, 0x0, 0x0, …)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/base/bucket.go:118 +0x8a
github.com/couchbase/sync_gateway/base.(*CouchbaseBucket).WriteUpdate(0xc82006a4d0, 0xc8521abde0, 0x17, 0x0, 0xc8bbf69830, 0x0, 0x0)
:100 +0xf3
github.com/couchbase/sync_gateway/db.(*Database).updateDoc(0xc8521ab7c0, 0xc8521abde0, 0x17, 0x0, 0xc82e967680, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:535 +0x54b
github.com/couchbase/sync_gateway/db.(*Database).PutExistingRev(0xc8521ab7c0, 0xc8521abde0, 0x17, 0xc8b189db30, 0xc8a2a2b480, 0x2, 0x2, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:513 +0x466
github.com/couchbase/sync_gateway/rest.(*handler).handleBulkDocs(0xc8866f58c0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/bulk_api.go:422 +0x1dc8
github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc8866f58c0, 0xf82f08, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:200 +0x7b4
github.com/couchbase/sync_gateway/rest.makeHandler.func1(0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:90 +0x77
net/http.HandlerFunc.ServeHTTP(0xc82036ed00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc8205ad950, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/gorilla/mux/mux.go:86 +0x2a6
github.com/couchbase/sync_gateway/rest.wrapRouter.func1(0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/routing.go:299 +0x32d
net/http.HandlerFunc.ServeHTTP(0xc820453140, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc82056fb00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc8a164a9a0)
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6

@alexegli,
Is the doc is CB about 20MB? if so you hit the limit.

Are all the conflicting branches marked as deleted:true?

The total doc size with the _sync section is about 10MB. I don’t see a deleted array in _sync and none of the doc versions in bodyMap have “deleted”:true. I can’t check previous revisions because they get cleaned up pretty quickly so they’ll all be gone from the DB by now. We don’t have any special conflict handling in our couchbase lite app, we just let the auto-conflict handling of couchbase lite work, where it chooses longest branch or most recent revision I think.

How does the 1000 revisions limit work in sync gateway? Shouldn’t only the most recent 1000 entries be in bodymap, revs, channels, etc? I don’t want our docs growing infinitely with every revision.

revs_limit=1000 means the rev tree won’t get deeper than 1000 revisions, but it doesn’t stop that rev tree from branching outwards with conflicting revisions.

You’ll have to tombstone these conflicts/non-winning branches if you don’t want to keep them around.

https://github.com/couchbase/sync_gateway/wiki/Rev-Tree-Storage-on-Couchbase-Server#pruning

687474703a2f2f636f756368626173652d6d6f62696c652e73332e616d617a6f6e6177732e636f6d2f696d616765732f72657674726565735f7072756e696e675f70726f626c656d2e706e67

https://github.com/couchbase/sync_gateway/wiki/Rev-Tree-Storage-on-Couchbase-Server#ways-in-which-rev-trees-can-blow-up-in-size

687474703a2f2f636f756368626173652d6d6f62696c652e73332e616d617a6f6e6177732e636f6d2f696d616765732f72657674726565735f626c6f7775705f726576626f64792e706e67

I had a similar issue on a doc with about 7000 revs, but it had no conflicts. Without any conflicts shouldn’t it just have 1000 entries in the _sync metadata? I’m not really concerned about the revision docs being stored, I know they get deleted very quickly because whenever we’ve tried to access a previous revision of a doc to look into why a conflict happened it’s always returned as being deleted. I’m more concerned about the _sync object itself growing infinitely just keeping an array of all the previous revisions.
Are you saying that sync gateway will grow the _sync object infinitely for docs that are updated often regardless of if they have conflicts or not?

I’m still confused about tombstones. Are they just the deleted versions on my db? E.g. when I try to find a previous version of a document and just get back { error: not_found, reason: deleted }.

I checked out the tombstone documentation (https://docs.couchbase.com/sync-gateway/current/managing-tombstones.html) but it sounds like it was written for sync gateway 1.5. In sync gateway 1.3.1 do I just need to manually purge the db once in while with the purge endpoint? I’m not actually that concerned about tombstones if they are just those deleted stub documents, I’m more concerned with the growth of the _sync object on our live documents. Will purging affect the live docs and clean up the _sync object inside them?