JSON error (str vs bytes) in cbbackup and cbbackupwrapper


#1

I’ve been experimenting with backing up a CB server from the command line.
On a fresh ubuntu 16.04 VM (with python3 installed and not python2), after cloning the couchbase-cli repo and installing pre-requisites, I’m getting the following error:

...
File "/home/ubuntu/couchbase-cli/pump_dcp.py", line 743, in total_msgs
reason="total_msgs", verify=opts.no_ssl_verify, cacert=opts.cacert)
File "/home/ubuntu/couchbase-cli/pump.py", line 1048, in rest_request_json
return None, rest_json, json.loads(rest_json)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

This occurs in both pump.py and cbbackupwrapper. cbbackupwrapper, line 199, I see:

data = json.loads(response.read())

It seems that the data coming back from the /pools/default/buckets url are bytes rather than strings. By changing the line to:

data = json.loads(response.read().decode("utf-8"))

everything works again. Similarly for pump.py on line 1048.
Is my server configuration screwy that could be causing bytes to be returned? (p.s. For a proper solution, I probably should check the type of the result before decoding…)

Side Note: all the scripts expect /usr/bin/python. But only /usr/bin/python3 exists on a clean ubuntu 16.04 instance in AWS. Creating an alias doesn’t help because of the #! usage in CLI scripts. An ugly workaround is to: ln -s /usr/bin/python3 /usr/bin/python. This is usually frowned upon because you dont want python2 scripts accidentally running python3, but 2 is not installed.


#2

Hello @paulr,

It’s great to see you cloning the repo and trying out the bleeding edge. We are in the middle of moving our code from Python2 to Python3.

  • Which SHA is your repo on?
  • Are you building Couchbase too?

From a quick glance at the code it looks like you have found a case where we not yet moved over to Python 3.

Is my server configuration screwy that could be causing bytes to be returned?

Pretty sure the server is all good and the issue is with couchbase-cli.

Side Note: all the scripts expect /usr/bin/python. But only /usr/bin/python3 exists on a clean ubuntu 16.04 instance in AWS. Creating an alias doesn’t help because of the #! usage in CLI scripts. An ugly workaround is to: ln -s /usr/bin/python3 /usr/bin/python . This is usually frowned upon because you dont want python2 scripts accidentally running python3, but 2 is not installed.

In the future we are changing the way the python code is packaged and will not be using the system python to avoid these issues.


#3

@paulr

I just tested this using the latest version of couchbase-cli and it works:

 ./cbbackupwrapper http://localhost:9000 ~/backup-2 -u Administrator -p asdasd
Waiting for the backup to complete...
SUCCESSFULLY COMPLETED!

It would be good to know more about the environment you hit this in.