Sync Gateway in Docker doesn't start from wait-for-it.sh

rominf · January 12, 2019, 8:07pm

I’m trying to use CouchBase Server and Sync Gateway using Docker and Docker Compose. Here is my docker-compose.yml file:

version: "3"
services:
couchbase-server:
    image: couchbase/server
    ports:
    - "8091:8091"
    - "11210:11210"
    volumes:
    - /opt/couchbase/var:/opt/couchbase/var
sync-gateway:
    image: couchbase/sync-gateway
    ports:
    - "4984:4984"
    - "4985:4985"
    volumes:
    - ./sync-gateway-config.json:/etc/sync_gateway/config.json
    - ./wait-for-it.sh:/wait-for-it.sh
    depends_on:
    - couchbase-server
    entrypoint: ['/wait-for-it.sh', 'couchbase-server:8091', '--timeout=64', '--', '/entrypoint.sh', '/etc/sync_gateway/config.json']

Here is my sync-gateway-config.json:

{
  "log": ["HTTP+"],
  "adminInterface":":4985",
  "interface": ":4984",
  "databases": {
    "db": {
      "server": "http://couchbase-server:8091",
      "bucket": "default",
      "username": "sync-gateway",
      "password": "password",
      "num_index_replicas": 0,
      "users": {
        "GUEST": {"disabled": false, "admin_channels": ["*"] }
      }
    }
  }
}

Here what I get on “docker-compose up”:

Starting couchbase-docker-compose_couchbase-server_1 ... 
e[1Ae[2K
Starting couchbase-docker-compose_couchbase-server_1 ... e[32mdonee[0m
e[1BStarting couchbase-docker-compose_sync-gateway_1     ... 
e[1Ae[2K
Starting couchbase-docker-compose_sync-gateway_1     ... e[32mdonee[0m
e[1BAttaching to couchbase-docker-compose_couchbase-server_1, couchbase-docker-compose_sync-gateway_1
sync-gateway_1      | wait-for-it.sh: waiting 64 seconds for couchbase-server:8091
couchbase-server_1  | Starting Couchbase Server -- Web UI available at http://<ip>:8091
couchbase-server_1  | and logs available in /opt/couchbase/var/lib/couchbase/logs
sync-gateway_1      | wait-for-it.sh: couchbase-server:8091 is available after 12 seconds
sync-gateway_1      | 2019-01-12T20:05:29.118Z [WRN] Using deprecated config option: log. Use logging.console.log_keys instead. -- rest.(*ServerConfig).deprecatedConfigLoggingFallback() at config.go:623
sync-gateway_1      | 2019-01-12T20:05:29.118Z [ERR] No logFilePath configured, and --defaultLogFilePath flag is not set. Log files required for product support are not being generated. -- base.(*LoggingConfig).Init() at logging_config.go:40
sync-gateway_1      | 2019-01-12T20:05:29.118Z ==== Couchbase Sync Gateway/2.1.2(2;35fe28e) ====
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] Console LogKeys: [HTTP HTTP+]
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] Console LogLevel: info
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] Log Redaction Level: none
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] requestedSoftFDLimit < currentSoftFdLimit (5000 < 1048576) no action needed
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] Opening db /db as bucket "default", pool "default", server <http://couchbase-server:8091>
sync-gateway_1      | 2019-01-12T20:05:29.118Z [INF] GoCBCustomSGTranscoder Opening Couchbase database default on <http://couchbase-server:8091> as user "sync-gateway"
sync-gateway_1      | 2019-01-12T20:05:30.318Z [INF] Successfully opened bucket
sync-gateway_1      | 2019-01-12T20:05:32.821Z [ERR] Error opening database db: operation has timed out
sync-gateway_1      | Unrecoverable GoCB error calling Stats()
sync-gateway_1      | github.com/couchbase/sync_gateway/base.(*CouchbaseBucketGoCB).CouchbaseServerVersion
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket_gocb.go:2254
sync-gateway_1      | github.com/couchbase/sync_gateway/base.IsMinimumServerVersion
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket.go:546
sync-gateway_1      | github.com/couchbase/sync_gateway/rest.(*ServerContext)._getOrAddDatabaseFromConfig
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:477
sync-gateway_1      | github.com/couchbase/sync_gateway/rest.(*ServerContext).getOrAddDatabaseFromConfig
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:301
sync-gateway_1      | github.com/couchbase/sync_gateway/rest.(*ServerContext).AddDatabaseFromConfig
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:755
sync-gateway_1      | github.com/couchbase/sync_gateway/rest.RunServer
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:902
sync-gateway_1      | github.com/couchbase/sync_gateway/rest.ServerMain
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:964
sync-gateway_1      | main.main
sync-gateway_1      | 	/home/couchbase/jenkins/workspace/sgw-unix-build/2.1.2/enterprise/godeps/src/github.com/couchbase/sync_gateway/main.go:40
sync-gateway_1      | runtime.main
sync-gateway_1      | 	/usr/local/go/1.8.5/go/src/runtime/proc.go:185
sync-gateway_1      | runtime.goexit
sync-gateway_1      | 	/usr/local/go/1.8.5/go/src/runtime/asm_amd64.s:2197 -- rest.RunServer() at config.go:903
couchbase-docker-compose_sync-gateway_1 exited with code 1

I can access the default bucket from web UI without any problems. What should I do?

rominf · January 15, 2019, 6:59am

I found that the problem was because of usage of wait-for-it.sh. I removed and sync-gateway starts.

I want to use it because sync-gateway starts almost instantly and tries to connect to the couchbase-server, which starts much slower. I get lots of errors Error opening bucket: failed to connect to any of the specified hosts. Sometimes sync-gateway gives up and terminates.
I cannot understand the reason, why doesn’t it work. wait-for-it.sh just waits and launches the entry point script. What could possibly go wrong?

bbrks · January 15, 2019, 11:20am

I think the main issue here is using wait-for-it.sh. Couchbase Server starts listening on its ports before the cluster is fully initialized and ready to accept bucket ops.

A more intelligent wait script would query the Couchbase API for readiness and block SG startup until that point.

/pools/default/buckets -> Make sure all “status” properties on each bucket are “healthy”.
https://docs.couchbase.com/server/6.0/rest-api/rest-endpoints-all.html

rominf · January 15, 2019, 12:43pm

Thank you!

I wrote a simple script that does that you’ve recommended. It works. Here it is:

#!/usr/bin/env bash

couchbase_sync_gateway_config_file_path=/etc/sync_gateway/config.json
couchbase_server_url=`cat $couchbase_sync_gateway_config_file_path | grep '"server":' | grep -o '"http://.*"' | sed 's/"//g'`

while ! { curl -X GET -u $1 $couchbase_server_url/pools/default/buckets -H "accept: application/json" -s | grep -q '"status":"healthy"'; }; do
  sleep 1
done

/entrypoint.sh $couchbase_sync_gateway_config_file_path

You have to pass login and password to it in the format “user:password”.

What do you think?

I think it’s worth to make it better and put it in the codebase so that others do not reinvent the wheel.

UPD0: What I don’t like is that passwords are stored in two files:

It’s a security issue.
I have dev and prod environments. So, I have to duplicate files.

UPD1: Solved UPD0:
Now wait-for-couchbase-server.sh looks:

#!/usr/bin/env bash

sync_gateway_config_file_path=/etc/sync_gateway/config.json
sync_gateway_config_template_file_path="${sync_gateway_config_file_path}.template"

sed "s/{{ sync_gateway_user }}/$1/g;s/{{ sync_gateway_password }}/$2/g" $sync_gateway_config_template_file_path > $sync_gateway_config_file_path

couchbase_server_url=`cat $sync_gateway_config_file_path | grep '"server":' | grep -o '"http://.*"' | sed 's/"//g'`

sync_gateway_auth="$1:$2"
while ! { curl -X GET -u $sync_gateway_auth $couchbase_server_url/pools/default/buckets -H "accept: application/json" -s | grep -q '"status":"healthy"'; }; do
  sleep 1
done

/entrypoint.sh $sync_gateway_config_file_path

And docker-compose.yml:

version: "3"
services:
  couchbase-server:
    image: couchbase/server
    ports:
    - "8091:8091"
    - "11210:11210"
    volumes:
    - /opt/couchbase/var:/opt/couchbase/var
  sync-gateway:
    image: couchbase/sync-gateway
    ports:
    - "4984:4984"
    - "4985:4985"
    volumes:
    - ./sync-gateway-config.json:/etc/sync_gateway/config.json.template
    - ./wait-for-couchbase-server.sh:/wait-for-couchbase-server.sh
    depends_on:
    - couchbase-server
    entrypoint: ["/wait-for-couchbase-server.sh", "${sync_gateway_user}", "${sync_gateway_password}"]

And sync-gateway-config.json:

{
  "log": ["HTTP+"],
  "adminInterface":":4985",
  "interface": ":4984",
  "databases": {
    "db": {
      "server": "http://couchbase-server:8091",
      "bucket": "default",
      "username": "{{ sync_gateway_user }}",
      "password": "{{ sync_gateway_password }}",
      "num_index_replicas": 0,
      "enable_shared_bucket_access": true,                                                                                        
      "import_docs": true, 
      "users": {
        "GUEST": {"disabled": false, "admin_channels": ["*"] }
      },
      "sync": `function (doc, oldDoc) {                                                                                           
          requireAdmin();                                                                                                         
      }`
    }
  }
}

You have to set environment variables sync_gateway_user and sync_gateway_password before starting docker-compose.

UPD2: See https://github.com/couchbase/sync_gateway/issues/3915

John2 · July 4, 2019, 3:04am

Hi all,

I got the similar error when I starting up the sync gateway under VM environment, can anyone help what’s wrong for my setup?

Sync gateway : 2.5.0
Couchbase Server: 6.0

2019-07-03T22:44:12.949-04:00 [WRN] Using deprecated config option: "log". Use "logging.console.log_keys" instead. -- rest.(*ServerConfig).deprecatedConfigLoggingFallback.func5() at config.go:674
2019-07-03T22:44:12.949-04:00 [INF] Logging: Console to stderr
2019-07-03T22:44:12.949-04:00 [INF] Logging: Files disabled
2019-07-03T22:44:12.949-04:00 [ERR] No log_file_path property specified in config, and --defaultLogFilePath command line flag was not set. Log files required for product support are not being generated. -- base.(*LoggingConfig).Init.func1() at logging_config.go:62
2019-07-03T22:44:12.949-04:00 [INF] Logging: Console level: info
2019-07-03T22:44:12.949-04:00 [INF] Logging: Console keys: [* HTTP]
2019-07-03T22:44:12.949-04:00 [INF] Logging: Redaction level: none
2019-07-03T22:44:12.949-04:00 [INF] requestedSoftFDLimit >= currentHardFdLimit (5000 >= 4096) capping at 4096
2019-07-03T22:44:12.949-04:00 [INF] Configured process to allow 4096 open file descriptors
2019-07-03T22:44:12.949-04:00 [INF] Logging stats with frequency: 1m0s
2019-07-03T22:44:12.949-04:00 [INF] Opening db /beer-sample as bucket "beer-sample", pool "default", server <http://192.168.225.140:8091>
2019-07-03T22:44:12.949-04:00 [INF] GoCBCustomSGTranscoder Opening Couchbase database beer-sample on <http://192.168.225.140:8091> as user "sync_user"
2019-07-03T22:44:12.950-04:00 [INF] Auth: Attempting credential authentication http://192.168.225.140:8091?http_idle_conn_timeout=90000&http_max_idle_conns=64000&http_max_idle_conns_per_host=256&n1ql_timeout=75000
2019-07-03T22:44:12.959-04:00 [INF] Successfully opened bucket beer-sample
2019-07-03T22:44:12.959-04:00 [INF] Set query timeouts for bucket beer-sample to cluster:1m15s, bucket:1m15s
2019-07-03T22:44:15.459-04:00 [ERR] Error opening database beer-sample: operation has timed out
Unrecoverable GoCB error calling Stats()
github.com/couchbase/sync_gateway/base.(*CouchbaseBucketGoCB).CouchbaseServerVersion
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket_gocb.go:2262
github.com/couchbase/sync_gateway/base.IsMinimumServerVersion
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/base/bucket.go:331
github.com/couchbase/sync_gateway/rest.(*ServerContext)._getOrAddDatabaseFromConfig
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:482
github.com/couchbase/sync_gateway/rest.(*ServerContext).getOrAddDatabaseFromConfig
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:306
github.com/couchbase/sync_gateway/rest.(*ServerContext).AddDatabaseFromConfig
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/server_context.go:813
github.com/couchbase/sync_gateway/rest.RunServer
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:964
github.com/couchbase/sync_gateway/rest.ServerMain
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/rest/config.go:1085
main.main
	/home/couchbase/jenkins/workspace/sgw-unix-build/2.5.0/enterprise/godeps/src/github.com/couchbase/sync_gateway/main.go:25
runtime.main
	/usr/local/go/1.11.5/go/src/runtime/proc.go:201
runtime.goexit
	/usr/local/go/1.11.5/go/src/runtime/asm_amd64.s:1333 -- rest.RunServer() at config.go:965

And sync-gateway-config.json :

{
	"log": ["*"],
	"interface": ":4984",
	"adminInterface": ":4985",
	"databases": {
		"beer-sample": {
			"server": "http://192.168.225.140:8091",
			"users": {
				"GUEST": {
					"disabled": false,
					"admin_channels": ["*"]
				}
			},
			"bucket": "beer-sample",
			"username": "XXXXX",
			"password": "XXXXX",
			"allow_conflicts": false,
			"revs_limit": 20
		}
	}
}