Why Sync Gateway will crash?

Suddenly the sync gateway crashes with strange messages

@carl.yau please provide Sync Gateway logs showing the error messages up to the point of the crash.

goroutine 1159 [IO wait]:
net.(*pollDesc).Wait(0xc0820abaf0, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc0820ab9e0, 0xafa820, 0x7, 0xda13d8, 0xc, 0x
0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc0820ab980, 0xc082b0c140, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc0821367b0, 0xc082b0c140, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
io.ReadAtLeast(0x333e80, 0xc0821367b0, 0xc082b0c140, 0x18, 0x18, 0x18, 0x0, 0x0,
0x0)
c:/go/src/io/io.go:298 +0xf8
io.ReadFull(0x333e80, 0xc0821367b0, 0xc082b0c140, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/io/io.go:316 +0x74
github.com/couchbase/gomemcached.(*MCRequest).Receive(0xc082a20060, 0x333e80, 0x
c0821367b0, 0xc082b0c140, 0x18, 0x18, 0xc082a73360, 0x0, 0x0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xf5
github.com/couchbase/gomemcached/client.(*Client).runFeed(0xc082b122d0, 0xc082a8
4e40, 0xc082b0f3c0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:270 +0x133
created by github.com/couchbase/gomemcached/client.(*Client).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:251 +0x2e6

goroutine 1186 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc0825c7d80, 0xc0
82afd080, 0xc082512d80, 0xc0826c69c0, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 912 [sleep]:
github.com/couchbase/sync_gateway/db.func┬╖003()
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/change_cache.go:113 +0x6d
created by github.com/couchbase/sync_gateway/db.(*changeCache).Init
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/change_cache.go:115 +0x378

goroutine 1113 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc0822f2900, 0xc0
823ccec0, 0xc08236ad80, 0xc0825ec340, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 1093 [chan receive]:
github.com/couchbase/sync_gateway/db.(*DatabaseContext).watchDocChanges(0xc08224
81c0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/assimilator.go:16 +0xd7
created by github.com/couchbase/sync_gateway/db.NewDatabaseContext
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/database.go:120 +0x662

goroutine 1094 [IO wait]:
net.(*pollDesc).Wait(0xc0820aaef0, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc0820aade0, 0xafa820, 0x7, 0xda13d8, 0x49d,
0x0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc0820aad80, 0xc082571e80, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc0823b80a0, 0xc082571e80, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
io.ReadAtLeast(0x333e80, 0xc0823b80a0, 0xc082571e80, 0x18, 0x18, 0x18, 0x0, 0x0,
0x0)
c:/go/src/io/io.go:298 +0xf8
io.ReadFull(0x333e80, 0xc0823b80a0, 0xc082571e80, 0x18, 0x18, 0x49d, 0x0, 0x0)
c:/go/src/io/io.go:316 +0x74
github.com/couchbase/gomemcached.(*MCRequest).Receive(0xc08236a840, 0x333e80, 0x
c0823b80a0, 0xc082571e80, 0x18, 0x18, 0xc0825f2f19, 0x0, 0x0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xf5
github.com/couchbase/gomemcached/client.(*Client).runFeed(0xc0824acfc0, 0xc0822e
f0e0, 0xc082571e40)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:270 +0x133
created by github.com/couchbase/gomemcached/client.(*Client).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:251 +0x2e6

goroutine 1095 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc08236c500, 0xc0
82571e40, 0xc08203ca80, 0xc0822d9a20, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 1085 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).run(0xc0825c7d80)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:58 +0x3ff
created by github.com/couchbase/go-couchbase.(*Bucket).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:42 +0x229

goroutine 1096 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).run(0xc0822f2900)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:58 +0x3ff
created by github.com/couchbase/go-couchbase.(*Bucket).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:42 +0x229

goroutine 1097 [chan receive]:
github.com/couchbase/sync_gateway/base.func┬╖002()
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/base/bucket.go:120 +0x74
created by github.com/couchbase/sync_gateway/base.CouchbaseBucket.StartCouchbase
TapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/base/bucket.go:130 +0x281

goroutine 1098 [chan receive]:
github.com/couchbase/sync_gateway/db.(*Shadower).readTapFeed(0xc0822f2940)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/shadower.go:60 +0xa9
created by github.com/couchbase/sync_gateway/db.NewShadower
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/shadower.go:36 +0x1e6

goroutine 1099 [IO wait]:
net.(*pollDesc).Wait(0xc082556470, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc082556360, 0xafa820, 0x7, 0xda13d8, 0x18, 0
x0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc082556300, 0xc0824e3000, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc0823b80a8, 0xc0824e3000, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
io.ReadAtLeast(0x333e80, 0xc0823b80a8, 0xc0824e3000, 0x18, 0x18, 0x18, 0x0, 0x0,
0x0)
c:/go/src/io/io.go:298 +0xf8
io.ReadFull(0x333e80, 0xc0823b80a8, 0xc0824e3000, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/io/io.go:316 +0x74
github.com/couchbase/gomemcached.(*MCRequest).Receive(0xc082905e60, 0x333e80, 0x
c0823b80a8, 0xc0824e3000, 0x18, 0x18, 0x1117fc0, 0x0, 0x0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xf5
github.com/couchbase/gomemcached/client.(*Client).runFeed(0xc08249d020, 0xc0824f
0de0, 0xc0824e2fc0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:270 +0x133
created by github.com/couchbase/gomemcached/client.(*Client).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:251 +0x2e6

goroutine 1100 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc0822f2900, 0xc0
824e2fc0, 0xc08236ad80, 0xc0825ec320, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 1101 [IO wait]:
net.(*pollDesc).Wait(0xc0825568f0, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc0825567e0, 0xafa820, 0x7, 0xda13d8, 0x2f5,
0x0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc082556780, 0xc0824e3580, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc0823b80d0, 0xc0824e3580, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
io.ReadAtLeast(0x333e80, 0xc0823b80d0, 0xc0824e3580, 0x18, 0x18, 0x18, 0x0, 0x0,
0x0)
c:/go/src/io/io.go:298 +0xf8
io.ReadFull(0x333e80, 0xc0823b80d0, 0xc0824e3580, 0x18, 0x18, 0x0, 0x0, 0x0)
c:/go/src/io/io.go:316 +0x74
github.com/couchbase/gomemcached.(*MCRequest).Receive(0xc082af2360, 0x333e80, 0x
c0823b80d0, 0xc0824e3580, 0x18, 0x18, 0xc08206ef19, 0x0, 0x0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/mc_req.go:130 +0xf5
github.com/couchbase/gomemcached/client.(*Client).runFeed(0xc08249d200, 0xc0824f
1380, 0xc0824e3540)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:270 +0x133
created by github.com/couchbase/gomemcached/client.(*Client).StartTapFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/gomemcached/client/tap_feed.go:251 +0x2e6

goroutine 1102 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc0822f2900, 0xc0
824e3540, 0xc08236ad80, 0xc0825ec360, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 1144 [select]:
github.com/couchbase/go-couchbase.(*TapFeed).forwardTapEvents(0xc082742c00, 0xc0
828c46a0, 0xc08272c180, 0xc08262b8e0, 0x13)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:104 +0x306
created by github.com/couchbase/go-couchbase.(*TapFeed).connectToNodes
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/go-couchbase/tap.go:96 +0x43e

goroutine 1120 [IO wait]:
net.(*pollDesc).Wait(0xc082619c70, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc082619b60, 0xafa820, 0x7, 0xda13d8, 0x0, 0x
0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc082619b00, 0xc0829cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc08238b9b0, 0xc0829cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
net/http.noteEOFReader.Read(0x333e80, 0xc08238b9b0, 0xc0820b8c08, 0xc0829cd000,
0x1000, 0x1000, 0x9b72e0, 0x0, 0x0)
c:/go/src/net/http/transport.go:1270 +0x75
net/http.(*noteEOFReader).Read(0xc0829c8ec0, 0xc0829cd000, 0x1000, 0x1000, 0xc08
2017600, 0x0, 0x0)
:125 +0xdb
bufio.(*Reader).fill(0xc082718de0)
c:/go/src/bufio/bufio.go:97 +0x1d5
bufio.(*Reader).Peek(0xc082718de0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
c:/go/src/bufio/bufio.go:132 +0xf7
net/http.(*persistConn).readLoop(0xc0820b8bb0)
c:/go/src/net/http/transport.go:842 +0xab
created by net/http.(*Transport).dialConn
c:/go/src/net/http/transport.go:660 +0xca6

goroutine 1201 [select]:
net/http.(*persistConn).writeLoop(0xc0820b8bb0)
c:/go/src/net/http/transport.go:945 +0x424
created by net/http.(*Transport).dialConn
c:/go/src/net/http/transport.go:661 +0xcc3

goroutine 1203 [select]:
net/http.(*persistConn).writeLoop(0xc0820b8840)
c:/go/src/net/http/transport.go:945 +0x424
created by net/http.(*Transport).dialConn
c:/go/src/net/http/transport.go:661 +0xcc3

goroutine 1146 [IO wait]:
net.(*pollDesc).Wait(0xc082ab88f0, 0x72, 0x0, 0x0)
c:/go/src/net/fd_poll_runtime.go:84 +0x4e
net.(*ioSrv).ExecIO(0xc082136008, 0xc082ab87e0, 0xafa820, 0x7, 0xda13d8, 0x0, 0x
0, 0x0)
c:/go/src/net/fd_windows.go:188 +0x305
net.(*netFD).Read(0xc082ab8780, 0xc0829d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
c:/go/src/net/fd_windows.go:470 +0x180
net.(*conn).Read(0xc08238b968, 0xc0829d9000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
c:/go/src/net/net.go:121 +0xe3
net/http.noteEOFReader.Read(0x333e80, 0xc08238b968, 0xc082132688, 0xc0829d9000,
0x1000, 0x1000, 0x9b72e0, 0x0, 0x0)
c:/go/src/net/http/transport.go:1270 +0x75
net/http.(*noteEOFReader).Read(0xc082553f80, 0xc0829d9000, 0x1000, 0x1000, 0xc08
2014000, 0x0, 0x0)
:125 +0xdb
bufio.(*Reader).fill(0xc082593ce0)
c:/go/src/bufio/bufio.go:97 +0x1d5
bufio.(*Reader).Peek(0xc082593ce0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
c:/go/src/bufio/bufio.go:132 +0xf7
net/http.(*persistConn).readLoop(0xc082132630)
c:/go/src/net/http/transport.go:842 +0xab
created by net/http.(*Transport).dialConn
c:/go/src/net/http/transport.go:660 +0xca6

goroutine 1147 [select]:
net/http.(*persistConn).writeLoop(0xc082132630)
c:/go/src/net/http/transport.go:945 +0x424
created by net/http.(*Transport).dialConn
c:/go/src/net/http/transport.go:661 +0xcc3

goroutine 1187 [semacquire]:
sync.(*Cond).Wait(0xc082154500)
c:/go/src/sync/cond.go:62 +0xa5
github.com/couchbase/sync_gateway/db.(*changeListener).Wait(0xc082248020, 0xc082
a68380, 0x8, 0x8, 0x2, 0x0)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/change_listener.go:117 +0x2ab
github.com/couchbase/sync_gateway/db.(*changeWaiter).Wait(0xc082ccf600, 0xc0824f
0c60)
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/change_listener.go:178 +0x4d
github.com/couchbase/sync_gateway/db.func┬╖014()
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/changes.go:419 +0x1790
created by github.com/couchbase/sync_gateway/db.(*Database).MultiChangesFeed
c:/jenkins/workspace/sync-gateway-windows-builds/app-under-test/sync_gat
eway/src/github.com/couchbase/sync_gateway/db/changes.go:468 +0x5d6

C:\Program Files (x86)\Couchbase>

Before that. There is a lot of errors messages of these

23:33:40.415966 2017-02-19T23:33:40.415+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:40.419966 2017-02-19T23:33:40.419+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:40.956997 2017-02-19T23:33:40.956+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:40.965997 2017-02-19T23:33:40.965+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:40.998999 2017-02-19T23:33:40.998+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.088004 2017-02-19T23:33:41.088+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.091004 2017-02-19T23:33:41.091+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.093004 2017-02-19T23:33:41.093+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.738041 2017-02-19T23:33:41.738+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.740041 2017-02-19T23:33:41.740+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.743042 2017-02-19T23:33:41.743+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at shadower.go:79
23:33:41.784044 2017-02-19T23:33:41.784+08:00 WARNING: Error applying change from external bucket: 409 Not imported – db.(*Shadower).readTapFeed() at s

@carl.yau Unfortunately the above logs don’t include the full crash information, but unless this is a configuration issue, this looks like a bug.

Can you file a ticket at https://github.com/couchbase/sync_gateway/issues, and include a link to the full crash log (as a gist), as well as your Sync Gateway config?

Thanks!

@carl.yau,

Looks like you are using shadow buckets. That functionality will be deprecated soon so it is advised to do all CRUD through Sync Gateway REST.

Deprecation notices

Bucket shadowing is being deprecated in 1.4 and will be unsupported in an upcoming version (2.x) of Couchbase Mobile. The recommended approach to perform operations on the bucket dedicated to Couchbase Mobile is to use the Sync Gateway REST API.
source: https://developer.couchbase.com/documentation/mobile/1.4/whatsnew.html