Possible bug: Running cbbackupmgr with docker


#1

Hello!

I have a working installation of Couchbase Server Enterprise Edition 5.5.1 build 3511 running within a Docker container. I’m able to access the dashboard from localhost:8091, and have a cluster running with two other instances in their own docker containers, all within a docker network. Now, I’m attempting to create a backup of this cluster using cbbackupmgr from the host OS (Windows). However, the backup fails due to a tcp connection timing out. The full log file is as follows:

2018-09-05T12:45:09.464-04:00 (Cmd) cbbackupmgr version 1.3.0
2018-09-05T12:45:09.467-04:00 (Cmd) backup --archive C:\tmp\backup --repo cluster --cluster'couchbase://localhost --username <ud>Administrator</ud> --password ******** --purge
2018-09-05T12:45:09.509-04:00 (Rest) GET http://localhost:8091/pools 200
2018-09-05T12:45:09.777-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.799-04:00 (Plan) Executing transfer plan
2018-09-05T12:45:09.799-04:00 (Plan) Checking for data movement restrictions between travel-sample and travel-sample
2018-09-05T12:45:09.805-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:09.829-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.865-04:00 (Rest) GET http://localhost:8091/pools/default 200
2018-09-05T12:45:09.867-04:00 Transfering from Couchbase Server 5.5.1
2018-09-05T12:45:09.924-04:00 (Plan) Transfering bucket configuration for travel-sample to travel-sample
2018-09-05T12:45:09.959-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:09.972-04:00 (Plan) Transfering views definitions for travel-sample to travel-sample
2018-09-05T12:45:10.002-04:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2018-09-05T12:45:10.012-04:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
2018-09-05T12:45:10.809-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:11.811-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:12.813-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:13.817-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:14.819-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:15.821-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:16.823-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:17.826-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:18.829-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:19.831-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:20.834-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:21.836-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:22.840-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:23.843-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:24.845-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:25.847-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:26.849-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:27.852-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:28.854-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:29.857-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:30.859-04:00 Got error moving cursor in terminal: The operation completed successfully.
2018-09-05T12:45:31.017-04:00 ERRO: dial tcp 172.18.0.2:8091: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. -- plan.(*views).execute() at views.go:58
2018-09-05T12:45:31.017-04:00 (Plan) Transfer plan failed due to error dial tcp 172.18.0.2:8091: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
2018-09-05T12:45:31.018-04:00 (Cmd) Error backing up cluster: dial tcp 172.18.0.2:8091: connectex: A' connection attempt failed because the connected party did not properly respond after a period of time, or' established connection failed because connected host has failed to respond.

172.18.0.2:8091 is the IP address of the node at localhost:8091 on the internal Docker network. It looks like cbbackupmgr is successfully connecting to the node and grabbing what buckets exist, but when it actually goes to create the archive, it’s using the self-reported IP from the server, not the IP from the CLI invocation.

I can use “docker exec -i -t /bin/bash” to login into the docker container, and use the cbbackupmgr within the Docker container to create a backup of the cluster, using a Docker mount location to transfer the archive back to the host OS filesystem. However, initiating a backup from the host OS fails. Is this a bug?


#2

Hi @ryarlagadda,

You are correct, cbbackupmgr first uses the passed in address of localhost:8091 to pull information from the server before actually backing up your data. The information it pulls contains the IP addresses of each node in the cluster. In 5.5.0+ this cluster information will contain the internal address and possible alternate addresses for each node. The cbbackupmgr tool is grabbing the internal IP address (172.18.0.2) which is not routable from the host machine. Unfortunately this network setup is non-standard for Couchbase and cbbackupmgr does not yet support using alternate addresses that the Couchbase instance provides. There is a work around, however. Simple way would be to docker exec into the Couchbase container, run cbbackupmgr, zip up the backup directory, then docker copy it out of the container back to the host machine.