Can too many revisions on a document cause sync gateway or couchbase server to crash?

Couchbase Server Community 4.1.1
Sync Gateway Community 1.3.1

Sync gateway crashed for us recently and it caused a crash of our couchbase node as well with. Right before the error in the sync gateway logs I saw it write to a doc with 2820 revisons. We have no max revisions set, so the default max of 1000 should be applied, but in the _sync object in the doc I can see it has all revisions listed in history and records of conflicts in bodymap for about 4500 revision conflicts. Could writing to this huge doc have caused a crash? Is there anyway to force sync gateway to discard old revision info in the _sync object so it only stores about a 100 revisions?

This log message was in the couchbase server logs:
Service ‘memcached’ exited with status 137. Restarting. Messages: 2019-09-09T17:14:01.461912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) stream created!
2019-09-09T17:14:01.462115Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 41) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 121415 was last seqno sent, reason: The stream ended due to all items being streamed
2019-09-09T17:14:01.463876Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Creating stream with start seqno 129890 and end seqno 129891
2019-09-09T17:14:01.463912Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) stream created!
2019-09-09T17:14:01.464091Z WARNING (todos) DCP (Producer) eq_dcpq:mapreduce_view: todos _design/sync_gateway (prod/main) - (vb 203) Stream closing, 0 items sent from backfill phase, 1 items sent from memory phase, 129891 was last seqno sent, reason: The stream ended due to all items being streamed

Sync gateway log error message:
http: panic serving runtime error: invalid memory address or nil pointer dereference
goroutine 41825347 [running]:
net/http.(*conn).serve.func1(0xc8a164a9a0, 0x7fd4bc33ef30, 0xc868698b38)
/usr/local/go/src/net/http/server.go:1287 +0xb5*Client).CASNext(0xc828d5b290, 0xc8b78301b4, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0xc8b78387f0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x15e*Bucket).casNext.func2(0xc828d5b290, 0xf801b4, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x7b*Bucket).Do.func1(0xc82039a700, 0xc8000001b4, 0xc8bc1a4cb8, 0xc8202ac500, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x127*Bucket).Do(0xc82039a700, 0xc8521abde0, 0x17, 0xc8bc1a4cb8, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x27c*Bucket).casNext(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4d40, 0x89bd00)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x12b*Bucket).update(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4df0, 0xf7eea1, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x85*Bucket).WriteUpdate(0xc82039a700, 0xc8521abde0, 0x17, 0x0, 0xc8bc1a4e50, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0xba, 0xc820191060, 0x15, 0xd45280, 0x7, 0xc82026a770, 0x5, 0x0, 0x0, 0x0, …)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x8a*CouchbaseBucket).WriteUpdate(0xc82006a4d0, 0xc8521abde0, 0x17, 0x0, 0xc8bbf69830, 0x0, 0x0)
:100 +0xf3*Database).updateDoc(0xc8521ab7c0, 0xc8521abde0, 0x17, 0x0, 0xc82e967680, 0x0, 0x0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x54b*Database).PutExistingRev(0xc8521ab7c0, 0xc8521abde0, 0x17, 0xc8b189db30, 0xc8a2a2b480, 0x2, 0x2, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x466*handler).handleBulkDocs(0xc8866f58c0, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x1dc8*handler).invoke(0xc8866f58c0, 0xf82f08, 0x0, 0x0)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x7b4, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x77
net/http.HandlerFunc.ServeHTTP(0xc82036ed00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a*Router).ServeHTTP(0xc8205ad950, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x2a6, 0xc8a164aa50, 0xc84dc37180)
/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/ +0x32d
net/http.HandlerFunc.ServeHTTP(0xc820453140, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc82056fb00, 0x7fd4bc341d20, 0xc8a164aa50, 0xc84dc37180)
/usr/local/go/src/net/http/server.go:1862 +0x19e
/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:1910 +0x3f6

Is the doc is CB about 20MB? if so you hit the limit.

Are all the conflicting branches marked as deleted:true?

The total doc size with the _sync section is about 10MB. I don’t see a deleted array in _sync and none of the doc versions in bodyMap have “deleted”:true. I can’t check previous revisions because they get cleaned up pretty quickly so they’ll all be gone from the DB by now. We don’t have any special conflict handling in our couchbase lite app, we just let the auto-conflict handling of couchbase lite work, where it chooses longest branch or most recent revision I think.

How does the 1000 revisions limit work in sync gateway? Shouldn’t only the most recent 1000 entries be in bodymap, revs, channels, etc? I don’t want our docs growing infinitely with every revision.

revs_limit=1000 means the rev tree won’t get deeper than 1000 revisions, but it doesn’t stop that rev tree from branching outwards with conflicting revisions.

You’ll have to tombstone these conflicts/non-winning branches if you don’t want to keep them around.



I had a similar issue on a doc with about 7000 revs, but it had no conflicts. Without any conflicts shouldn’t it just have 1000 entries in the _sync metadata? I’m not really concerned about the revision docs being stored, I know they get deleted very quickly because whenever we’ve tried to access a previous revision of a doc to look into why a conflict happened it’s always returned as being deleted. I’m more concerned about the _sync object itself growing infinitely just keeping an array of all the previous revisions.
Are you saying that sync gateway will grow the _sync object infinitely for docs that are updated often regardless of if they have conflicts or not?

I’m still confused about tombstones. Are they just the deleted versions on my db? E.g. when I try to find a previous version of a document and just get back { error: not_found, reason: deleted }.

I checked out the tombstone documentation ( but it sounds like it was written for sync gateway 1.5. In sync gateway 1.3.1 do I just need to manually purge the db once in while with the purge endpoint? I’m not actually that concerned about tombstones if they are just those deleted stub documents, I’m more concerned with the growth of the _sync object on our live documents. Will purging affect the live docs and clean up the _sync object inside them?