Error Running dbtransfer Tool


#1

I’m running cb server version 5.0 on ubuntu servers. When I’m trying to run this cbtransfer command to export some data to a csv file:

./cbtransfer \
-b metrics-metadata \
--key=metrics\.content_partner:.* \
-u Administrator \
-p password \
http://server_host:8091 \
/mnt/storage/content_partner.csv

I got error message about “AttributeError: ‘module’ object has no attribute ‘loads’” (complete error message shown below). I checked the python versions on the server and we do have python 2.7 and 3.4 installed. Can anyone help with this?

  Traceback (most recent call last):
  File "/opt/couchbase/lib/python/cbtransfer", line 33, in <module>
    pump_transfer.exit_handler(pump_transfer.Transfer().main(sys.argv))
  File "/opt/couchbase/lib/python/pump_transfer.py", line 80, in main
    rv = pumpStation.run()
  File "/opt/couchbase/lib/python/pump.py", line 112, in run
    rv, source_map, sink_map = self.check_endpoints()
  File "/opt/couchbase/lib/python/pump.py", line 167, in check_endpoints
    rv, source_map = self.source_class.check(self.opts, self.source_spec)
  File "/opt/couchbase/lib/python/pump_dcp.py", line 73, in check
    err, map = pump.rest_couchbase(opts, spec)
  File "/opt/couchbase/lib/python/pump.py", line 1025, in rest_couchbase
    result, errors = rest.list_buckets(True)
  File "/opt/couchbase/lib/python/cluster_manager.py", line 671, in list_buckets
    result, errors = self._get(url)
  File "/opt/couchbase/lib/python/cluster_manager.py", line 32, in g
    return f(*args, **kwargs)
  File "/opt/couchbase/lib/python/cluster_manager.py", line 1260, in _get
    return _handle_response(response, self.debug)
  File "/opt/couchbase/lib/python/cluster_manager.py", line 1320, in _handle_response
    return response.json(), None
  File "/opt/couchbase/lib/python/requests/models.py", line 799, in json
    return complexjson.loads(
AttributeError: 'module' object has no attribute 'loads'

#2

Hi @hclc,

Would you mind sharing a sample document or two? I’d like to try and reproduce.


#3

Hi,

I’m not sure if that’s document related, as this error message seems like the python script couldn’t run.

Here’s the document samples:
Key:
metrics.content_partner:4K MEDIA

document:

{
  "akamaiCpCode": "81845",
  "platformCpId": 50000369,
  "metadata-key": "4K MEDIA", 
  "ingestDirectory": "4kmedia",
  "companyName": "4K MEDIA INC.",
  "supplierNumber": "3338",
  "revSharePercent": -0.01,  
  "buybackRevSharePercent": -0.01,
  "contentPartnerName": "NS CP 4K Media",
  "cpIdentifier": "4K Media",
  "revCalculationType": 0,
  "contentPartnerId": 650,
  "holdMusicLicenseFee": 0,
  "asperaPassword": "ABCPASSWORD",
  "metadata-type": "metrics.content_partner",
  "isDeleted": 0,
  "allowBuyback": 0,
  "parentContentPartnerName": "Third Party",
  "firstRelease": "2013-02-28 00:00:00.0",
  "status": 0
}

#4

Seemed to work okay when I tried it (but I was using 5.5 in Windows). I created a ticket, you may want to keep an eye on it: https://issues.couchbase.com/browse/MB-30980


#5

my bucket has over 600 million documents in it, this command filters out one set of keys from them. does that affect the result of this command?


#6

The error is coming from the requests module which cbtransfer uses. Depending on the version of Python being used either simplejson or json module is imported as complexjson. The stack trace is saying that the module loaded does not have the loads function.

To help us investigate this further can you please answer the following questions:

  • What version of Ubuntu is being used?
  • What version of Python is the script being executed as? The output of python -v should be enough to answer this.

#7

Thank @pvarley. Here’s the output with the information you need:

Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.16.0-77-generic x86_64)

 * Documentation:  https://help.ubuntu.com/
Last login: Sun Aug 12 11:35:42 2018 from 10.86.65.30
charles.chao@las-metrics-couchbase-01:~$ sudo su
root@las-metrics-couchbase-01:/home/CORP/charles.chao# python -v
# installing zipimport hook
import zipimport # builtin
# installed zipimport hook
# /usr/lib/python2.7/site.pyc matches /usr/lib/python2.7/site.py
import site # precompiled from /usr/lib/python2.7/site.pyc
# /usr/lib/python2.7/os.pyc matches /usr/lib/python2.7/os.py
import os # precompiled from /usr/lib/python2.7/os.pyc
import errno # builtin
import posix # builtin
# /usr/lib/python2.7/posixpath.pyc matches /usr/lib/python2.7/posixpath.py
import posixpath # precompiled from /usr/lib/python2.7/posixpath.pyc
# /usr/lib/python2.7/stat.pyc matches /usr/lib/python2.7/stat.py
import stat # precompiled from /usr/lib/python2.7/stat.pyc
# /usr/lib/python2.7/genericpath.pyc matches /usr/lib/python2.7/genericpath.py
import genericpath # precompiled from /usr/lib/python2.7/genericpath.pyc
# /usr/lib/python2.7/warnings.pyc matches /usr/lib/python2.7/warnings.py
import warnings # precompiled from /usr/lib/python2.7/warnings.pyc
# /usr/lib/python2.7/linecache.pyc matches /usr/lib/python2.7/linecache.py
import linecache # precompiled from /usr/lib/python2.7/linecache.pyc
# /usr/lib/python2.7/types.pyc matches /usr/lib/python2.7/types.py
import types # precompiled from /usr/lib/python2.7/types.pyc
# /usr/lib/python2.7/UserDict.pyc matches /usr/lib/python2.7/UserDict.py
import UserDict # precompiled from /usr/lib/python2.7/UserDict.pyc
# /usr/lib/python2.7/_abcoll.pyc matches /usr/lib/python2.7/_abcoll.py
import _abcoll # precompiled from /usr/lib/python2.7/_abcoll.pyc
# /usr/lib/python2.7/abc.pyc matches /usr/lib/python2.7/abc.py
import abc # precompiled from /usr/lib/python2.7/abc.pyc
# /usr/lib/python2.7/_weakrefset.pyc matches /usr/lib/python2.7/_weakrefset.py
import _weakrefset # precompiled from /usr/lib/python2.7/_weakrefset.pyc
import _weakref # builtin
# /usr/lib/python2.7/copy_reg.pyc matches /usr/lib/python2.7/copy_reg.py
import copy_reg # precompiled from /usr/lib/python2.7/copy_reg.pyc
# /usr/lib/python2.7/traceback.pyc matches /usr/lib/python2.7/traceback.py
import traceback # precompiled from /usr/lib/python2.7/traceback.pyc
# /usr/lib/python2.7/sysconfig.pyc matches /usr/lib/python2.7/sysconfig.py
import sysconfig # precompiled from /usr/lib/python2.7/sysconfig.pyc
# /usr/lib/python2.7/re.pyc matches /usr/lib/python2.7/re.py
import re # precompiled from /usr/lib/python2.7/re.pyc
# /usr/lib/python2.7/sre_compile.pyc matches /usr/lib/python2.7/sre_compile.py
import sre_compile # precompiled from /usr/lib/python2.7/sre_compile.pyc
import _sre # builtin
# /usr/lib/python2.7/sre_parse.pyc matches /usr/lib/python2.7/sre_parse.py
import sre_parse # precompiled from /usr/lib/python2.7/sre_parse.pyc
# /usr/lib/python2.7/sre_constants.pyc matches /usr/lib/python2.7/sre_constants.py
import sre_constants # precompiled from /usr/lib/python2.7/sre_constants.pyc
# /usr/lib/python2.7/_sysconfigdata.pyc matches /usr/lib/python2.7/_sysconfigdata.py
import _sysconfigdata # precompiled from /usr/lib/python2.7/_sysconfigdata.pyc
# /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc matches /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.py
import _sysconfigdata_nd # precompiled from /usr/lib/python2.7/plat-x86_64-linux-gnu/_sysconfigdata_nd.pyc
# /usr/lib/python2.7/sitecustomize.pyc matches /usr/lib/python2.7/sitecustomize.py
import sitecustomize # precompiled from /usr/lib/python2.7/sitecustomize.pyc
import encodings # directory /usr/lib/python2.7/encodings
# /usr/lib/python2.7/encodings/__init__.pyc matches /usr/lib/python2.7/encodings/__init__.py
import encodings # precompiled from /usr/lib/python2.7/encodings/__init__.pyc
# /usr/lib/python2.7/codecs.pyc matches /usr/lib/python2.7/codecs.py
import codecs # precompiled from /usr/lib/python2.7/codecs.pyc
import _codecs # builtin
# /usr/lib/python2.7/encodings/aliases.pyc matches /usr/lib/python2.7/encodings/aliases.py
import encodings.aliases # precompiled from /usr/lib/python2.7/encodings/aliases.pyc
# /usr/lib/python2.7/encodings/utf_8.pyc matches /usr/lib/python2.7/encodings/utf_8.py
import encodings.utf_8 # precompiled from /usr/lib/python2.7/encodings/utf_8.pyc
Python 2.7.6 (default, Oct 26 2016, 20:30:19)
[GCC 4.8.4] on linux2
Type "help", "copyright", "credits" or "license" for more information.
dlopen("/usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so", 2);
import readline # dynamically loaded from /usr/lib/python2.7/lib-dynload/readline.x86_64-linux-gnu.so

#8

Thank you for the output. I have tried this on Couchbase Server 5.0.0 EE on Ubuntu 14.04.5 LTS and it works for me. Is this a normal install using the package?

I suspect something in the environment is causing the problem as the json module used by cbtransfer is the standard module that comes with Python.

Can you run two more commands please:

bash -x /opt/couchbase/bin/cbtransfer -b metrics-metadata --key=metrics\.content_partner:.* -u Administrator -p password http://server_host:8091 /mnt/storage/content_partner.csv

and then

LD_LIBRARY_PATH=/opt/couchbase/lib PYTHONPATH=/opt/couchbase/lib/python python -v /opt/couchbase/lib/python/cbtransfer -b metrics-metadata '--key=metrics.content_partner:.*' -u Administrator -p password http://server_host:8091 /home/vagrant/content_partner.csv 2>python.log

Please share the python.log file created, this will let us know what modules from where are being imported.


#9

Hi, @pvarley,

Thanks for your response. I was OOO for a week, but when I tried your first command, it actually worked - there’s no python error anymore and the command appeared to be working.

However, the new problem is, this command seems print out every key it skips during the process, like this:

2018-08-28 11:18:00,607: w3 skipping msg with key: metrics.******

This bucket I’m working on has over 700 million keys. If we print out everything skipped it might take too much time. Is there a way to tune the logging level of this command?

Thanks,

Charles


#10

@hclc The --silent option will disable the logging of this message. Only error messages will be logged to the console.