Sync gateway 2.7.3 cannot use couchbase schema to connect to couchbase server

Mobile Sync Gateway
1

Environment

Autonomous Operator

couchbase server version: 6.5.1 enterprise

sync gateway version: 2.7.3 enterprise

Sync Gateway Config

"databases": {
      "foo": {
        "server": "couchbase://couchbase.couchbase.svc",
......

couchbase.couchbase.svc is k8s service domain which can be accessed.

When I use couchbase schema, sync gateway can not start and come out some errors:

2020-06-03T01:30:31.848Z [ERR] gocb: memdClient read failure: EOF -- base.GoCBCoreLogger.Log() at logger_external.go:66
2020-06-03T01:30:31.848Z [WRN] gocb: Failed to close authentication client (close tcp 172.22.0.53:32966->172.22.0.41:8091: use of closed network connection) -- base.GoCBCoreLogger.Log() at logger_external.go:68

And I have to change http schema to fix the issue

"databases": {
      "foo": {
        "server": "http://couchbase.couchbase.svc:8091",
......

PS. It works well in 2.6.0 version.

2

Are there any more logs besides the two posted? I’d expect to see more logging before the gocb EOF error that might explain why it fails.

3

@bbrks

2020-06-04T01:49:23.856Z ==== Couchbase Sync Gateway/2.7.3(3;33d352f) EE ====
2020-06-04T01:49:23.856Z [INF] Logging: Console to stderr
2020-06-04T01:49:23.856Z [INF] Logging: Files to /var/tmp/sglogs
2020-06-04T01:49:23.856Z [INF] Logging: Console level: debug
2020-06-04T01:49:23.856Z [INF] Logging: Console keys: [* HTTP]
2020-06-04T01:49:23.856Z [INF] Logging: Redaction level: none
2020-06-04T01:49:23.856Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
2020-06-04T01:49:23.856Z [INF] Logging stats with frequency: 1m0s
2020-06-04T01:49:23.856Z [INF] Opening db /foo as bucket "foo", pool "default", server <couchbase://couchbase.couchbase.svc>
2020-06-04T01:49:23.856Z [INF] GoCBCustomSGTranscoder Opening Couchbase database foo on <couchbase://couchbase.couchbase.svc> as user "***"
2020-06-04T01:49:23.857Z [INF] Auth: Attempting credential authentication couchbase://couchbase.couchbase.svc?http_idle_conn_timeout=90000&http_max_idle_conns=64000&http_max_idle_conns_per_host=256&kv_pool_size=2&n1ql_timeout=75000&operation_tracing=false
2020-06-04T01:49:23.870Z [ERR] gocb: memdClient read failure: EOF -- base.GoCBCoreLogger.Log() at logger_external.go:66
2020-06-04T01:49:23.871Z [WRN] gocb: Failed to close authentication client (close tcp 172.22.0.59:41874->172.22.0.41:8091: use of closed network connection) -- base.GoCBCoreLogger.Log() at logger_external.go:68
2020-06-04T01:49:23.871Z [INF] Error opening bucket foo: failed to connect to any of the specified hosts
2020-06-04T01:49:23.871Z [DBG] RetryLoop retrying Attempt to connect to bucket : foo after 5 ms.
4

@bbrks It works well in version 2.6.0 with the same config.

2020-06-04T02:09:44.969Z ==== Couchbase Sync Gateway/2.6.0(127;b4c828d) EE ====
2020-06-04T02:09:44.969Z [INF] Logging: Console to stderr
2020-06-04T02:09:44.969Z [INF] Logging: Files to /var/tmp/sglogs
2020-06-04T02:09:44.969Z [INF] Logging: Console level: debug
2020-06-04T02:09:44.969Z [INF] Logging: Console keys: [* HTTP]
2020-06-04T02:09:44.969Z [INF] Logging: Redaction level: none
2020-06-04T02:09:44.969Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
2020-06-04T02:09:44.969Z [INF] Logging stats with frequency: 1m0s
2020-06-04T02:09:44.969Z [INF] Opening db /foo as bucket "foo", pool "default", server <couchbase://couchbase.couchbase.svc>
2020-06-04T02:09:44.970Z [INF] GoCBCustomSGTranscoder Opening Couchbase database foo on <couchbase://couchbase.couchbase.svc> as user "***"
2020-06-04T02:09:44.970Z [INF] Auth: Attempting credential authentication couchbase://couchbase.couchbase.svc?http_idle_conn_timeout=90000&http_max_idle_conns
=64000&http_max_idle_conns_per_host=256&n1ql_timeout=75000
2020-06-04T02:09:44.976Z [INF] Successfully opened bucket foo
2020-06-04T02:09:44.981Z [INF] Set query timeouts for bucket foo to cluster:1m15s, bucket:1m15s
2020-06-04T02:09:44.981Z [INF] Initializing indexes with numReplicas: 0...
2020-06-04T02:09:45.007Z [INF] Query: Index sg_channels_x1 doesn't exist, creating...

sync gateway config

{
    "interface":":4984",
    "adminInterface":":4985",
    "logging": {
      "log_file_path": "/var/tmp/sglogs",
      "console": {
          "log_level": "debug",
          "log_keys": ["*"],
          "color_enabled": true
      },
      "error": {
          "enabled": true,
          "rotation": {
              "max_size": 100,
              "max_age": 360,
              "localtime": false
          }
      },
      "warn": {
          "enabled": true,
          "rotation": {
              "max_size": 100,
              "max_age": 180,
              "localtime": false
          }
      },
      "info": {
          "enabled": true,
          "rotation": {
              "max_size": 100,
              "max_age": 6,
              "localtime": false
          }
      },
      "debug": {
          "enabled": false,
          "rotation": {
              "max_size": 100,
              "max_age": 2,
              "localtime": false
          }
      }
    },
    "databases": {
      "foo": {
        "server": "couchbase://couchbase.couchbase.svc",
        "bucket": "foo",
        "username": "***",
        "password": "***,",
        "users": {
          "GUEST": {"disabled": true}
        },
        "enable_shared_bucket_access": true,
        "import_docs": true,
        "allow_conflicts": false,
        "use_views": false,
        "num_index_replicas": 0,
        "bucket_op_timeout_ms": 5000,
        "sync": `
        function (doc, oldDoc) {
          }
        `
      }
    }
  }
5

That’s odd. This should work. Bootstrapping over http to post 8091 isn’t recommended. What version of CAO was used to deploy cluster?

What do the following commands output? (Follow instructions here)

kubectl exec -ti dnsutils -- nslookup -type=SRV _couchbases._tcp.couchbase.couchbase.svc

kubectl exec -ti dnsutils -- nslookup couchbase.couchbase.svc

Does connection work if you replace “server” entry with DNS node endpoint of individual data service node Instead of the SRV record?

6

CAO version

couchbase/operator:1.2.2

dnsutil output

root@dnsutils:/# nslookup -type=SRV _couchbases._tcp.couchbase.couchbase.svc
Server:         172.21.0.10
Address:        172.21.0.10#53

_couchbases._tcp.couchbase.couchbase.svc.couchbase.svc.cluster.local    canonical name = _couchbases._tcp.couchbase.couchbase.svc.cluster.local.
_couchbases._tcp.couchbase.couchbase.svc.cluster.local  service = 0 100 18091 couchbase-0001.couchbase.couchbase.svc.cluster.local.


root@dnsutils:/# nslookup couchbase.couchbase.svc
Server:         172.21.0.10
Address:        172.21.0.10#53

couchbase.couchbase.svc.couchbase.svc.cluster.local     canonical name = couchbase.couchbase.svc.cluster.local.
Name:   couchbase.couchbase.svc.cluster.local
Address: 172.22.0.41

Replace “server” entry

I change the “server” entry and it works.

"databases": {
      "foo": {
        "server": "couchbase://couchbase-0001.couchbase.couchbase.svc.cluster.local",

2020-06-05T01:52:52.961Z ==== Couchbase Sync Gateway/2.7.3(3;33d352f) EE ====
2020-06-05T01:52:52.961Z ==== Couchbase Sync Gateway/2.7.3(3;33d352f) EE ====
2020-06-05T01:52:52.961Z [INF] Logging: Console to stderr
2020-06-05T01:52:52.961Z [INF] Logging: Files to /var/tmp/sglogs
2020-06-05T01:52:52.961Z [INF] Logging: Console level: debug
2020-06-05T01:52:52.961Z [INF] Logging: Console keys: [* HTTP]
2020-06-05T01:52:52.961Z [INF] Logging: Redaction level: none
2020-06-05T01:52:52.961Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
2020-06-05T01:52:52.961Z [INF] Logging stats with frequency: 1m0s
2020-06-05T01:52:52.962Z [INF] Opening db /foo as bucket "foo", pool "default", server <couchbase://couchbase-0001.couchbase.couchbase.svc.cluster.local>
2020-06-05T01:52:52.962Z [INF] GoCBCustomSGTranscoder Opening Couchbase database foo on <couchbase://couchbase-0001.couchbase.couchbase.svc.cluster.local> as user "***"
2020-06-05T01:52:52.998Z [INF] Auth: Attempting credential authentication couchbase://couchbase-0001.couchbase.couchbase.svc.cluster.local?http_idle_conn_timeout=90000&http_max_idle_conns=64000&http_max_idle_conns_per_host=256&kv_pool_size=2&n1ql_timeout=75000&operation_tracing=false
2020-06-05T01:52:53.008Z [INF] Successfully opened bucket foo
7

There’s also something wrong with couchbase python SDK 2.5.12.

When I user couchbase schema, it gives out some error:

couchbase.exceptions.LCB_0x2D (generated, catch: CouchbaseNetworkError, CouchbaseTransientError): <RC=0x2D[The remote host closed the connection], There was a problem while trying to send/receive your request over the network. This may be a result of a bad network or a misconfigured client or server, C Source=(src/bucket.c,1071)>

However, I use node SDK 2.4.3, and set the url as couchbase://xxx, It works.

8

That is probably a question for the SDK team. Its unrelated to the sync gateway so please create a separate post and tag it as such so you can have appropriate folks looking at it

9

Hi @priya.rajagopal,

I have same issues in my eventing node.
I found this error:
er logs, err: EOF
2020-06-21T07:28:02.934+00:00 [Error] Consumer::getFailoverLogOpCallback [worker_xxxx_0:/tmp/127.0.0.1:8091_worker_xxxx_0.sock:0] Failed to get failover logs, err: EOF
2020-06-21T07:28:03.126+00:00 [Error] [gocb] memdClient read failure: EOF
2020-06-21T07:28:03.127+00:00 [Warn] [gocb] Failed to close authentication client (close tcp 10.131.15.193:41472->172.30.195.20:11210: use of closed network connection

10

I can confirm that it doesn’t matter if you connect over http:// or couchbase://

You’ll still be flooded with:
memdClient read failure: EOF – gocbcore%!e(MISSING)v7.logExf() at logging.go:144

I’m using Couchbase Sync Gateway 2.8.2 and Couchbase Server 6.6