Getting this error: "interface conversion: interface is nil, not string"

Hello. We’re getting this error on a Sync Gateway console when requesting a document with several conflicts from our mobile app, which crashes after getting the error. Does this look familiar to anyone?

2016-12-01T08:27:23.732-05:00 Changes+: MultiChangesFeed waiting...   (to doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:23.733-05:00 Changes+: Waiting for "zm_database"'s count to pass 153
2016-12-01T08:27:23.732-05:00 Changes: sending 1 change(s)
2016-12-01T08:27:23.808-05:00 HTTP:  #706: POST /zm_database/_bulk_get?revs=true&attachments=true  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:23.890-05:00 HTTP:  #707: POST /zm_database/_bulk_get?revs=true&attachments=true  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:24.414-05:00 HTTP:  #708: POST /zm_database/_bulk_get?revs=true&attachments=true  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:27.638-05:00 HTTP:  #709: PUT /zm_database/_local/95782a2f8cac657eaab4dc694e9386bb85063ecc  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:27.640-05:00 HTTP+: #709:     --> 201   (3.0 ms)
2016-12-01T08:27:29.407-05:00 HTTP:  #710: PUT /zm_database/_local/1166ff49b87bd73d756a6ce47244d139078bee45  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016-12-01T08:27:29.409-05:00 HTTP+: #710:     --> 201   (3.0 ms)
2016-12-01T08:27:30.402-05:00 CRUD+: No old revision "-9m2-0QlNOaGVH9qaEQUKnf" / "8-66567699bc9e2fb6c6b212a06107a0ee"
2016-12-01T08:27:30.761-05:00 HTTP+: #708:     --> 200   (6369.3 ms)
2016-12-01T08:27:31.918-05:00 HTTP:  #711: POST /zm_database/_bulk_get?revs=true&attachments=true  (as doc_4322ddd7515d4a55a1c6f03565015c97)
2016/12/01 08:27:32 http: panic serving 10.10.2.69:49758: interface conversion: interface is nil, not string
goroutine 1033 [running]:
net/http.(*conn).serve.func1(0xc088840000, 0x1d64030, 0xc082056f58)
        c:/go/src/net/http/server.go:1287 +0xbc
github.com/couchbase/sync_gateway/db.(*Database).loadBodyAttachments(0xc08ee8a9c0, 0xc08b98a540, 0x1, 0xc, 0x0, 0x0)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/attachment.go:123 +0x293
github.com/couchbase/sync_gateway/db.(*Database).GetRev(0xc08ee8a9c0, 0xc08ee8aac0, 0x17, 0xc08e75d0b0, 0x23, 0x1, 0xc08f13f350, 0x0, 0x0, 0xc08e75de90, ...)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/db/crud.go:166 +0xa78
github.com/couchbase/sync_gateway/rest.(*handler).handleBulkGet.func1(0xc08e75de30, 0x0, 0x0)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/bulk_api.go:364 +0x6dd
github.com/couchbase/sync_gateway/rest.(*handler).writeMultipart(0xc086796360, 0xd96c50, 0x5, 0xc08f13f798, 0x0, 0x0)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/handler.go:516 +0x3a1
github.com/couchbase/sync_gateway/rest.(*handler).handleBulkGet(0xc086796360, 0x0, 0x0)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/bulk_api.go:380 +0x49f
github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc086796360, 0xfd4d20, 0x0, 0x0)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/handler.go:201 +0x7bb
github.com/couchbase/sync_gateway/rest.makeHandler.func1(0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/handler.go:90 +0x7e
net/http.HandlerFunc.ServeHTTP(0xc0820364c0, 0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/go/src/net/http/server.go:1422 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc0823a2550, 0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/gorilla/mux/mux.go:86 +0x2ad
github.com/couchbase/sync_gateway/rest.wrapRouter.func1(0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/jenkins/workspace/sync-gateway-dev-build/app-under-test/sync_gateway/src/github.com/couchbase/sync_gateway/rest/routing.go:301 +0x334
net/http.HandlerFunc.ServeHTTP(0xc08230a320, 0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/go/src/net/http/server.go:1422 +0x41
net/http.serverHandler.ServeHTTP(0xc082327800, 0x3317c0, 0xc08206ac60, 0xc08ab95420)
        c:/go/src/net/http/server.go:1862 +0x1a5
net/http.(*conn).serve(0xc088840000)
        c:/go/src/net/http/server.go:1361 +0xbf5
created by net/http.(*Server).Serve
        c:/go/src/net/http/server.go:1910 +0x3fd

Thanks!

What version of Sync Gateway are you running?

I am getting the exact same error. I am running sync gateway v1.3.1. I know that some documents that had an _attachments object had that _attachments object removed, but I don’t know if those are the documents causing the error.

The call that seems to be consistently generating this error is for a specific revision of a specific document. I don’t understand why couchbase lite is requesting a specific document though and not just using _bulk_get. Here is the line that is always above the go error in our logs, but I modified the id and rev. NOTE: This rev had 3 conflicts, and the document is currently on rev 257:

2017-03-06T21:05:06.656Z HTTP:  #644580: GET /todos/c81c7d97-47c8-4152-jajsd-e0c4f5037386?rev=188-c20d8bbea275c8ebf552f95daksakd255f3e&revs=true&attachments=true&atts_since=%5B%22184-03a8cd14580a94dcd27e43a0bb71a913%22%2C%22176-62e7280c1bce7074e8e6b850cb1d3d63%22%2C%22172-cb18f190712554df3466b448db8ef8c6%22%2C%22146-e3c09a3fbc55dd764ff4a72386faf0da%22%2C%22140-25c3c718449158d4de91687057638568%22%2C%22137-b33f582e8513cda3f3fa4b331a70efae%22%2C%22130-88a1093a4360519020ce7c6a86e4ad66%22%2C%22120-ae0f836db9a04c95ff7d816c323abad8%22%2C%22116-a5d6cc3366818354f2d844ca532f2abb%22%2C%22104-f370a9ede870aed4de8c801c358c2a4d%22%5D  (as d23c3699-924c-sfsdfsb50e-51720cd2757a)
2017-03-06 21:05:06.665310 I | http: panic serving 127.0.0.1:39878: interface conversion: interface is nil, not string
goroutine 3129347 [running]:
net/http.(*conn).serve.func1(0xc83aaa1b80, 0x7fc6942e3d18, 0xc8391f3ce8)
	/usr/local/go/src/net/http/server.go:1287 +0xb5
github.com/couchbase/sync_gateway/db.(*Database).loadBodyAttachments(0xc834bc9e20, 0xc854d31e60, 0x1, 0xc83c5bcc30, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/attachment.go:152 +0x28c
github.com/couchbase/sync_gateway/db.(*Database).GetRevWithHistory(0xc834bc9e20, 0xc86f536f0b, 0x24, 0xc86f536f34, 0x24, 0x7fffffff, 0xc83c5bcc30, 0xa, 0xd, 0xc83c5bcc30, ...)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/db/crud.go:203 +0xce8
github.com/couchbase/sync_gateway/rest.(*handler).handleGetDoc(0xc8aa6986c0, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/doc_api.go:64 +0x490
github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc8aa6986c0, 0xf82fc8, 0x0, 0x0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:200 +0x7b4
github.com/couchbase/sync_gateway/rest.makeHandler.func1(0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:90 +0x77
net/http.HandlerFunc.ServeHTTP(0xc8202f6540, 0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/usr/local/go/src/net/http/server.go:1422 +0x3a
github.com/gorilla/mux.(*Router).ServeHTTP(0xc820481400, 0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/gorilla/mux/mux.go:86 +0x2a6
github.com/couchbase/sync_gateway/rest.wrapRouter.func1(0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/home/couchbase/jenkins/workspace/sgw-unix-build/1.3.1/community/godeps/src/github.com/couchbase/sync_gateway/rest/routing.go:299 +0x32d
net/http.HandlerFunc.ServeHTTP(0xc8203ed700, 0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/usr/local/go/src/net/http/server.go:1422 +0x3a
net/http.serverHandler.ServeHTTP(0xc82058eba0, 0x7fc697135b88, 0xc83aaa1c30, 0xc835b09ea0)
	/usr/local/go/src/net/http/server.go:1862 +0x19e
net/http.(*conn).serve(0xc83aaa1b80)
	/usr/local/go/src/net/http/server.go:1361 +0xbee
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:1910 +0x3f6

I think this is related to sync gateway issue: https://github.com/couchbase/sync_gateway/issues/1706

I did some more investigating and found that the url that causes this issue points to a document revision with an _attachments object, but no digest field. The original request that causes the 502 has url param attachments = true, but if I set it to attachments=false the request succeeds and I can see the broken _attachments object.

REQUEST: curl "localhost:4985/todos/<doc-id>?rev=<doc-rev>&revs=true&attachments=false&atts_since=[array of revs]"
RESPONSE: {"_attachments":{"profileImage":{"revpos":7,"stub":true}, ...}

We don’t touch the _attachments object directly anywhere in our code, and sync gateway wouldn’t let us save it even if we did so I think this is an issue with sync gateway. I can see in the bodymap field of the document that one of the revision bodymaps has digest missing from the _attachments object.

Also this user has 306 revisions currently, and the revision that is causing issues is rev 188. The bodymap has about 30 revs stored in it, and one of them has:
{\"_attachments\":{\"profileImage\":{\"revpos\":7,\"stub\":true}},"<doc body"}

My initial thought is that Sync Gateway shouldn’t be specifying “stub”:true when persisting the body of a conflicting revision to the bodymap - it should include the digest reference.

Filed https://github.com/couchbase/sync_gateway/issues/2378 to investigate further.

@alexegli If you’ve got any additional details on how to reproduce this scenario, please add them to the ticket.

Thanks, I’ll add them. The only thing I can think of though is to just create a lot of conflicts for a document with attachments, and some of the conflicts are to remove the attachments object altogether. I’ll update the ticket.

We are now also receiving this interface is nil panic:

2018-01-10 12:59:03.515647 I | http: panic serving x.x.x.x:54759: interface conversion: interface is nil, not string
  goroutine 835 [running]:
  net/http.(*conn).serve.func1(0xc420379000)
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:1491 +0x12a
  panic(0xae26e0, 0xc421eb79c0)
  #011/usr/local/go/1.7.4/go/src/runtime/panic.go:458 +0x243
  github.com/couchbase/sync_gateway/rest.(*handler).handleGetAttachment(0xc420e33200, 0xc421252500, 0x0)
  #011/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.5.0/community/godeps/src/github.com/couchbase/sync_gateway/rest/doc_api.go:155 +0x267
  github.com/couchbase/sync_gateway/rest.(*handler).invoke(0xc420e33200, 0xc5dec8, 0x0, 0x0)
  #011/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.5.0/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:221 +0x2c1
  github.com/couchbase/sync_gateway/rest.makeHandler.func1(0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.5.0/community/godeps/src/github.com/couchbase/sync_gateway/rest/handler.go:96 +0x80
  net/http.HandlerFunc.ServeHTTP(0xc420641f00, 0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:1726 +0x44
  github.com/gorilla/mux.(*Router).ServeHTTP(0xc420145040, 0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.5.0/community/godeps/src/github.com/gorilla/mux/mux.go:86 +0x204
  github.com/couchbase/sync_gateway/rest.wrapRouter.func1(0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/home/couchbase/jenkins/workspace/sgw-unix-build@2/1.5.0/community/godeps/src/github.com/couchbase/sync_gateway/rest/routing.go:297 +0x66f
  net/http.HandlerFunc.ServeHTTP(0xc420469a20, 0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:1726 +0x44
  net/http.serverHandler.ServeHTTP(0xc4203b5900, 0xf63dc0, 0xc4208ead00, 0xc42084f4a0)
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:2202 +0x7d
  net/http.(*conn).serve(0xc420379000, 0xf64980, 0xc420ac34c0)
 2018-01-10T12:59:03.496Z HTTP:  #3703: GET /
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:1579 +0x4b7
  created by net/http.(*Server).Serve
  #011/usr/local/go/1.7.4/go/src/net/http/server.go:2293 +0x44d

We are running {“couchdb”:“Welcome”,“vendor”:{“name”:“Couchbase Sync Gateway”,“version”:1.5},“version”:“Couchbase Sync Gateway/1.5.0(381;6a1598b)”}

We still have the instances running for further debugging in case a thread dump or something is needed.

It turned out we also had a document with an attachment without digest hash in the database. Recycling the Sync Gateway instances just lead to the same panic again. Manually removing the offending document and recyling the Sync Gateway instances once more fixed the issue.

Do you have any idea what is causing this and how it can be prevented in the future?

Any updates on this?
We encountered it again today and had to reboot the instances once more.

We got the same error again yesterday having to reboot the SG away. We are using v1.5.1 of the SG.