Sync Gateway gets stuck connecting to Couchbase Server 5.5

connections

#1

Hi there!

I’m hitting an issue when trying to migrate to Couchbase Server 5.5 with Sync Gateway 2.0, with the following message posted to the console as the last entry, followed by an indefinitely hang:

Using plain authentication for user (MY USERNAME)

I’ve tried both SG 2.0 as well as the current head of the master branch (508346b2dbc3419b1a6141a1ab039fbac31ec391), with the same symptoms. Any ideas welcome.

I create the bucket SG is connecting to as follows:

  echo "Creating ${APP_DATA_BUCKET} bucket..."
  couchbase-cli bucket-create -c http://127.0.0.1:8091 -u ${APP_ADMIN_USERNAME} -p ${APP_ADMIN_PASSWORD} \
  --bucket=${APP_DATA_BUCKET} \
  --bucket-type=couchbase \
  --bucket-ramsize=512 \
  --enable-index-replica=0

  couchbase-cli user-manage -c http://127.0.0.1:8091 \
    --username ${APP_ADMIN_USERNAME} \
    --password ${APP_ADMIN_PASSWORD} \
    --set \
    --rbac-username ${APP_DATA_BUCKET} \
    --rbac-password ${APP_COUCHBASE_RBAC_PASSWORD} \
    --roles="bucket_admin[${APP_DATA_BUCKET}],views_admin[${APP_DATA_BUCKET}],query_manage_index[${APP_DATA_BUCKET}],query_select[${APP_DATA_BUCKET}]" \
    --auth-domain local

I’ve added the various roles to the user based on the error messages, and also tried the map-reduce view based operation with same symptoms. – with 5.1 the only role I have there is bucket_admin[${APP_DATA_BUCKET}] with which I’ve had no problems connecting from SG to CB.

{
  "log": ["CRUD+","HTTP+","REST+", "Changes+", "Attach+"],
  "interface": ":<%- app_gateway_public_port %>",
  "adminInterface": ":<%- app_gateway_admin_port %>",
  "databases": {
      "<%- app_data_bucket %>": {
          "server":"http://<%- app_couchbase_hostname %>",
          "sync":`<%- sync_fn %>`,
          "send_www_authenticate_header": false,
          "use_views": false,
          "password": "<%- app_couchbase_rbac_password %>",
          "allow_conflicts": false,
          "num_index_replicas": 0
      }
  },
  "CORS": {
      "Origin": [<%- app_allowed_cors_origins.split(';').map(x => `"${x}"`).join(',') %>],
      "LoginOrigin": [<%- app_allowed_cors_origins.split(';').map(x => `"${x}"`).join(',') %>],
      "Headers": ["Content-Type"],
      "MaxAge": 17280000
  }
}

Matias


#2

Here’s a more complete log excerpt:

sync_gateway_1  | Bucket found.
sync_gateway_1  | Executing command...
sync_gateway_1  | 2018-08-02T01:13:45.586Z ==== Couchbase Sync Gateway/HEAD(nobranch)(508346b) ====
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Using deprecated config option: "log". Use "logging.console.log_keys" instead. -- rest.(*ServerConfig).deprecatedConfigLoggingFallback.func5() at config.go:656
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Deprecated log key: "CRUD+" found. Changing to: "CRUD". -- base.ToLogKey.func1() at log_keys.go:177
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Deprecated log key: "REST+" found. Changing to: "REST". -- base.ToLogKey.func1() at log_keys.go:177
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Invalid log key: REST+ -- base.ToLogKey.func2() at log_keys.go:187
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Deprecated log key: "Changes+" found. Changing to: "Changes". -- base.ToLogKey.func1() at log_keys.go:177
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Deprecated log key: "Attach+" found. Changing to: "Attach". -- base.ToLogKey.func1() at log_keys.go:177
sync_gateway_1  | 2018-08-02T01:13:45.586Z [WRN] Invalid log key: Attach+ -- base.ToLogKey.func2() at log_keys.go:187
sync_gateway_1  | 2018-08-02T01:13:45.586Z [ERR] No logFilePath configured, and --defaultLogFilePath flag is not set. Log files required for product support are not being generated. -- base.(*LoggingConfig).Init.func1() at logging_config.go:55
sync_gateway_1  | 2018-08-02T01:13:45.586Z [INF] Console LogKeys: [Changes CRUD HTTP HTTP+]
sync_gateway_1  | 2018-08-02T01:13:45.586Z [INF] Console LogLevel: info
sync_gateway_1  | 2018-08-02T01:13:45.587Z [INF] Log Redaction Level: none
sync_gateway_1  | 2018-08-02T01:13:45.587Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
sync_gateway_1  | 2018-08-02T01:13:45.587Z [INF] Opening db /manuscript_data as bucket "manuscript_data", pool "default", server <http://couchbase>
sync_gateway_1  | 2018-08-02T01:13:45.588Z [INF] GoCBCustomSGTranscoder Opening Couchbase database manuscript_data on <http://couchbase> as user "manuscript_data"
sync_gateway_1  | 2018-08-02T01:13:45.594Z [INF] Successfully opened bucket
sync_gateway_1  | 2018-08-02T01:13:45.597Z [INF] Initializing indexes with numReplicas: 0...
sync_gateway_1  | 2018-08-02T01:13:52.589Z [INF] Verifying index availability for bucket manuscript_data...
sync_gateway_1  | 2018-08-02T01:13:52.602Z [INF] Indexes ready for bucket manuscript_data.
sync_gateway_1  | 2018-08-02T01:13:52.602Z [INF] Initializing changes cache for database manuscript_data
sync_gateway_1  | 2018-08-02T01:13:52.672Z [INF] CBGoUtilsLogger: Using plain authentication for user <ud>manuscript_data</ud>

#3

… and more complete still with “*” used as log key:

sync_gateway_1  | Executing command...
sync_gateway_1  | 2018-08-02T01:43:34.202Z ==== Couchbase Sync Gateway/HEAD(nobranch)(508346b) ====
sync_gateway_1  | 2018-08-02T01:43:34.202Z [WRN] Using deprecated config option: "log". Use "logging.console.log_keys" instead. -- rest.(*ServerConfig).deprecatedConfigLoggingFallback.func5() at config.go:656
sync_gateway_1  | 2018-08-02T01:43:34.202Z [ERR] No logFilePath configured, and --defaultLogFilePath flag is not set. Log files required for product support are not being generated. -- base.(*LoggingConfig).Init.func1() at logging_config.go:55
sync_gateway_1  | 2018-08-02T01:43:34.202Z [INF] Console LogKeys: [* HTTP]
sync_gateway_1  | 2018-08-02T01:43:34.202Z [INF] Console LogLevel: info
sync_gateway_1  | 2018-08-02T01:43:34.202Z [INF] Log Redaction Level: none
sync_gateway_1  | 2018-08-02T01:43:34.202Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
sync_gateway_1  | 2018-08-02T01:43:34.202Z [INF] Opening db /library as bucket "library", pool "default", server <http://couchbase>
sync_gateway_1  | 2018-08-02T01:43:34.203Z [INF] GoCBCustomSGTranscoder Opening Couchbase database library on <http://couchbase> as user "library"
sync_gateway_1  | 2018-08-02T01:43:34.206Z [INF] Auth: Attempting credential authentication http://couchbase?http_idle_conn_timeout=90000&http_max_idle_conns=64000&http_max_idle_conns_per_host=256
sync_gateway_1  | 2018-08-02T01:43:34.210Z [INF] Successfully opened bucket
sync_gateway_1  | 2018-08-02T01:43:34.210Z [ERR] gocb: memdClient read failure: read tcp 172.25.0.3:42844->172.25.0.2:11210: use of closed network connection -- base.GoCBCoreLogger.Log() at logger_external.go:47
sync_gateway_1  | 2018-08-02T01:43:34.210Z [ERR] gocb: Failed to shut down client connection (close tcp 172.25.0.3:42844->172.25.0.2:11210: use of closed network connection) -- base.GoCBCoreLogger.Log() at logger_external.go:47
sync_gateway_1  | 2018-08-02T01:43:34.214Z [INF] Initializing indexes with numReplicas: 0...
sync_gateway_1  | 2018-08-02T01:43:34.439Z [INF] Query: Index sg_channels_1 doesn't exist, creating...
sync_gateway_1  | 2018-08-02T01:43:34.857Z [INF] Query: Index sg_channels_1 created successfully
sync_gateway_1  | 2018-08-02T01:43:35.007Z [INF] Query: Index sg_allDocs_1 doesn't exist, creating...
sync_gateway_1  | 2018-08-02T01:43:35.393Z [INF] Query: Index sg_allDocs_1 created successfully
sync_gateway_1  | 2018-08-02T01:43:35.484Z [INF] Query: Index sg_syncDocs_1 doesn't exist, creating...
sync_gateway_1  | 2018-08-02T01:43:35.932Z [INF] Query: Index sg_syncDocs_1 created successfully
sync_gateway_1  | 2018-08-02T01:43:36.024Z [INF] Query: Index sg_access_1 doesn't exist, creating...
sync_gateway_1  | 2018-08-02T01:43:36.410Z [INF] Query: Index sg_access_1 created successfully
sync_gateway_1  | 2018-08-02T01:43:36.498Z [INF] Query: Index sg_roleAccess_1 doesn't exist, creating...
sync_gateway_1  | 2018-08-02T01:43:36.904Z [INF] Query: Index sg_roleAccess_1 created successfully
sync_gateway_1  | 2018-08-02T01:43:36.996Z [INF] Query: Building deferred indexes: [sg_syncDocs_1 sg_roleAccess_1 sg_allDocs_1 sg_channels_1 sg_access_1]
sync_gateway_1  | 2018-08-02T01:43:41.060Z [INF] Verifying index availability for bucket library...
sync_gateway_1  | 2018-08-02T01:43:41.070Z [INF] Indexes ready for bucket library.
sync_gateway_1  | 2018-08-02T01:43:41.071Z [INF] Cache: Initializing changes cache with options {ChannelCacheOptions:{ChannelCacheMinLength:0 ChannelCacheMaxLength:0 ChannelCacheAge:0s} CachePendingSeqMaxWait:5s CachePendingSeqMaxNum:10000 CacheSkippedSeqMaxWait:1h0m0s}
sync_gateway_1  | 2018-08-02T01:43:41.071Z [INF] Initializing changes cache for database library
sync_gateway_1  | 2018-08-02T01:43:41.071Z [INF] DCP: Starting mutation feed on bucket library due to either channel cache mode or doc tracking (auto-import/bucketshadow)
sync_gateway_1  | 2018-08-02T01:43:41.071Z [INF] DCP: Using DCP feed for bucket: "library" (based on feed_type specified in config file)
sync_gateway_1  | 2018-08-02T01:43:41.144Z [INF] CBGoUtilsLogger: Using plain authentication for user <ud>library</ud>

#4

For reference, if anybody else stumbles upon this thread; the issue was caused by a combination of insufficient roles in Couchbase Server and Issue 3690 in Sync Gateway.

You will need to assign the Sync Gateway user the “Application Access” (bucket_full_access) role to fix.