Cbbackup script failing

lukeb · April 2, 2019, 3:25am

I’m trying to backup a bucket from couchbase but cbbackup is failing with this error (a thread gets an exception and then the script never finishes).

file.sh: Exception in thread w2:] 73.7% (8537/estimated 11590 msgs)
file.sh: Traceback (most recent call last):
file.sh:   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
file.sh:     self.run()
file.sh:   File "/usr/lib/python3.6/threading.py", line 864, in run
file.sh:     self._target(*self._args, **self._kwargs)
file.sh:   File "/couchbase-cli/pump.py", line 448, in run_worker
file.sh:     curx).run()
file.sh:   File "/couchbase-cli/pump.py", line 506, in run
file.sh:     rv_batch, batch = self.source.provide_batch()
file.sh:   File "/couchbase-cli/pump_dcp.py", line 254, in provide_batch
file.sh:     rv, batch = self.provide_dcp_batch_actual()
file.sh:   File "/couchbase-cli/pump_dcp.py", line 413, in provide_dcp_batch_actual
file.sh:     msg = (cmd, vbucket_id, key, flg, exp, cas, bytes([rev_seqno]), val, seqno, dtype, metalen,
file.sh: ValueError: bytes must be in range(0, 256)

I’m running it from a dockerfile which looks like this:

FROM gliderlabs/alpine
RUN apk-install bash
RUN apk add --update \
    python3 \
    python3-dev \
    py-pip \
    build-base \
    git \
	snappy \
	g++ \
	snappy-dev

EXPOSE 8080
ONBUILD RUN virtualenv /env && /env/bin/pip install -r /app/requirements.txt
RUN git clone https://github.com/couchbase/couchbase-cli.git
RUN pip3 install python-snappy
RUN pip3 install awscli
COPY file.sh /
RUN sed -i 's/\r$//' file.sh
ENTRYPOINT ["/bin/bash", "file.sh"]

where file.sh calls the backup script like this:

python3 couchbase-cli/cbbackup "$cbip" /backup -u "$cbuser" -p "$cbpass" -b "$cbbuck" -m "$backuptype" -v

We are running server version: Enterprise Edition 5.0.1 build 5003
cbbackup version is the latest.

Full log from the script (some paths omitted):

file.sh: 2019-04-02 03:15:38,685: mt cbbackup...
file.sh: 2019-04-02 03:15:38,686: mt  source : ---
file.sh: 2019-04-02 03:15:38,686: mt  sink   : /backup
file.sh: 2019-04-02 03:15:38,686: mt  opts   : {'bucket_source': '---', 'single_node': False, 'mode': '', 'id': None, 'key': None, 'vbucket_list': None, 'dry_run': False, 'username': '<xxx>', 'password': '<xxx>', 'username_dest': None, 'password_dest': None, 'ssl':False, 'no_ssl_verify': True, 'cacert': None, 'threads': 4, 'verbose': 1, 'silent': False, 'extra': {'batch_max_size': 1000.0, 'batch_max_bytes': 400000.0, 'cbb_max_mb': 100000.0, 'max_retry': 10.0, 'report': 5.0, 'report_full': 2000.0, 'recv_min_bytes': 4096.0, 'try_xwm': 1.0, 'nmv_retry': 1.0, 'rehash': 0.0, 'data_only': 0.0, 'design_doc_only': 0.0, 'conflict_resolve': 1.0, 'seqno': 0.0, 'mcd_compatible': 1.0, 'uncompress': 0.0, 'backoff_cap': 10.0, 'flow_control': 1.0, 'dcp_consumer_queue_length': 1000.0, 'allow_recovery_vb_remap': 0.0}, 'collection': None, 'force_txn': False}
file.sh: 2019-04-02 03:15:38,691: mt Starting new HTTP connection (1): ---
file.sh: 2019-04-02 03:15:39,128: mt bucket: ---
file.sh: 2019-04-02 03:16:31,707: w0   source : ---
file.sh: 2019-04-02 03:16:31,707: w0   sink   : /backup(---)
file.sh: 2019-04-02 03:16:31,707: w0          :                total |       last |    per sec
file.sh: 2019-04-02 03:16:31,707: w0    batch :                   63 |         63 |        1.2
file.sh: 2019-04-02 03:16:31,707: w0    byte  :             26809844 |   26809844 |   509981.0
file.sh: 2019-04-02 03:16:31,707: w0    msg   :                 3944 |       3944 |       75.0
file.sh: Exception in thread w2:] 73.7% (8537/estimated 11590 msgs)
file.sh: Traceback (most recent call last):
file.sh:   File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
file.sh:     self.run()
file.sh:   File "/usr/lib/python3.6/threading.py", line 864, in run
file.sh:     self._target(*self._args, **self._kwargs)
file.sh:   File "/couchbase-cli/pump.py", line 448, in run_worker
file.sh:     curx).run()
file.sh:   File "/couchbase-cli/pump.py", line 506, in run
file.sh:     rv_batch, batch = self.source.provide_batch()
file.sh:   File "/couchbase-cli/pump_dcp.py", line 254, in provide_batch
file.sh:     rv, batch = self.provide_dcp_batch_actual()
file.sh:   File "/couchbase-cli/pump_dcp.py", line 413, in provide_dcp_batch_actual
file.sh:     msg = (cmd, vbucket_id, key, flg, exp, cas, bytes([rev_seqno]), val, seqno, dtype, metalen,
file.sh: ValueError: bytes must be in range(0, 256)
file.sh: 2019-04-02 03:16:46,621: w1   source :---
file.sh: 2019-04-02 03:16:46,621: w1   sink   : /---
file.sh: 2019-04-02 03:16:46,621: w1          :                total |       last |    per sec
file.sh: 2019-04-02 03:16:46,621: w1    batch :                   74 |         74 |        1.1
file.sh: 2019-04-02 03:16:46,621: w1    byte  :             31811523 |   31811523 |   474734.3
file.sh: 2019-04-02 03:16:46,621: w1    msg   :                 3906 |       3906 |       58.3

thuan · April 3, 2019, 5:59am

Hi lukeb,
Thanks for using our product.
For Couchbase Server 5.0.x, cbbackup and cbrestore require python 2.6 or greater but not python 3.
If you have any question, drop me a line at thuan at couchbase dot com

lukeb · April 4, 2019, 7:31am

I’ve changed the python version in the container, it’s now 2.7.15

I get a different error now:

file.sh: Traceback (most recent call last):
file.sh:   File "couchbase-cli/cbbackup", line 8, in <module>
file.sh:     import pump_transfer
file.sh:   File "/couchbase-cli/pump_transfer.py", line 26
file.sh:     def exit_handler(err: Optional[str]):
file.sh:                         ^
file.sh: SyntaxError: invalid syntax

According to this: https://docs.python.org/3/library/typing.html
type annotations weren’t added until 3.5

Dockerfile:

FROM gliderlabs/alpine
RUN apk-install bash
RUN apk add --update \
    python \
    python-dev \
    py-pip \
    build-base \
    git \
	snappy \
	g++ \
	snappy-dev
EXPOSE 8080
ONBUILD RUN virtualenv /env && /env/bin/pip install -r /app/requirements.txt
RUN git clone https://github.com/couchbase/couchbase-cli.git
RUN pip install python-snappy
RUN pip install awscli
COPY file.sh /
RUN sed -i 's/\r$//' file.sh
ENTRYPOINT ["/bin/bash", "file.sh"]

script:

python -c 'import sys; print(sys.version_info[:])'
python couchbase-cli/cbbackup "$cbip" /backup -u "$cbuser" -p "$cbpass" -b "$cbbuck" -m "$backuptype" -v # run couchbases backup script

full output:

file.sh: (2, 7, 15, 'final', 0)
file.sh: Traceback (most recent call last):
file.sh:   File "couchbase-cli/cbbackup", line 8, in <module>
file.sh:     import pump_transfer
file.sh:   File "/couchbase-cli/pump_transfer.py", line 26
file.sh:     def exit_handler(err: Optional[str]):
file.sh:                         ^
file.sh:  SyntaxError: invalid syntax

lukeb · April 8, 2019, 3:00am

Looks like the type hints were added to pump_transfer.py a month ago, so cbbackup should be running on python 3 now.

So the issue is still my original error.

lukeb · April 9, 2019, 8:29am

Any update on this? We can’t backup our data right now.

lukeb · April 16, 2019, 3:01am

Running cbbackup in PyCharm, it fails the same way. Here are the logs:

2019-04-16 10:56:14,979: mt cbbackup...
2019-04-16 10:56:14,979: mt  source : http://couchbase********
2019-04-16 10:56:14,979: mt  sink   : /test1
2019-04-16 10:56:14,979: mt  opts   : {'bucket_source': '***', 'single_node': False, 'mode': 'full', 'id': None, 'key': None, 'vbucket_list': None, 'dry_run': False, 'username': '<xxx>', 'password': '<xxx>', 'username_dest': None, 'password_dest': None, 'ssl': False, 'no_ssl_verify': True, 'cacert': None, 'threads': 4, 'verbose': 1, 'silent': False, 'extra': {'batch_max_size': 1000.0, 'batch_max_bytes': 400000.0, 'cbb_max_mb': 100000.0, 'max_retry': 10.0, 'report': 5.0, 'report_full': 2000.0, 'recv_min_bytes': 4096.0, 'try_xwm': 1.0, 'nmv_retry': 1.0, 'rehash': 0.0, 'data_only': 0.0, 'design_doc_only': 0.0, 'conflict_resolve': 1.0, 'seqno': 0.0, 'mcd_compatible': 1.0, 'uncompress': 0.0, 'backoff_cap': 10.0, 'flow_control': 1.0, 'dcp_consumer_queue_length': 1000.0, 'allow_recovery_vb_remap': 0.0}, 'collection': None, 'force_txn': False}
2019-04-16 10:56:14,990: mt Starting new HTTP connection (1): couchbase*****
2019-04-16 10:56:15,651: mt bucket: ***
2019-04-16 10:57:26,465: w1   source : http://couchbase******(****@couchbase-node-2.node.consul:8091)
2019-04-16 10:57:26,466: w1   sink   : /test1(****@couchbase-node-2.node.consul:8091)
2019-04-16 10:57:26,466: w1          :                total |       last |    per sec
2019-04-16 10:57:26,467: w1    batch :                   89 |         89 |        1.3
2019-04-16 10:57:26,467: w1    byte  :             38534841 |   38534841 |   547829.2
2019-04-16 10:57:26,467: w1    msg   :                 6224 |       6224 |       88.5
2019-04-16 10:57:38,267: w0   source : http://couchbase****(*****@couchbase-node-1.node.consul:8091)
2019-04-16 10:57:38,268: w0   sink   : /test1(****@couchbase-node-1.node.consul:8091)
2019-04-16 10:57:38,268: w0          :                total |       last |    per sec
2019-04-16 10:57:38,269: w0    batch :                   76 |         76 |        0.9
2019-04-16 10:57:38,269: w0    byte  :             32069667 |   32069667 |   388204.7
2019-04-16 10:57:38,269: w0    msg   :                 6120 |       6120 |       74.1
Exception in thread w2:
Traceback (most recent call last):
  File "C:\Users\lukeb\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\lukeb\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Source\cb-cli\couchbase-cli\pump.py", line 446, in run_worker
    curx).run()
  File "C:\Source\cb-cli\couchbase-cli\pump.py", line 504, in run
    rv_batch, batch = self.source.provide_batch()
  File "C:\Source\cb-cli\couchbase-cli\pump_dcp.py", line 254, in provide_batch
    rv, batch = self.provide_dcp_batch_actual()
  File "C:\Source\cb-cli\couchbase-cli\pump_dcp.py", line 413, in provide_dcp_batch_actual
    msg = (cmd, vbucket_id, key, flg, exp, cas, bytes([rev_seqno]), val, seqno, dtype, metalen,
ValueError: bytes must be in range(0, 256)

Is anyone looking into this?

pvarley · April 16, 2019, 10:37am

Hello @lukeb

Out of interest why are you downloading from GitHub instead of using one of the prebuild package? Master is actively being developed for the next release and is expected to be unstable as it as not gone through the full range of testing yet.

If you still wish to download direct from GitHub I would recommend using the same major version as the cluster, in this case it would be 5.1.3. Please note that requires Python 2.6 as @thuan mentioned.

Exception in thread w2:
Traceback (most recent call last):
  File "C:\Users\lukeb\AppData\Local\Programs\Python\Python36\lib\threading.py", line 916, in _bootstrap_inner
    self.run()
  File "C:\Users\lukeb\AppData\Local\Programs\Python\Python36\lib\threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Source\cb-cli\couchbase-cli\pump.py", line 446, in run_worker
    curx).run()
  File "C:\Source\cb-cli\couchbase-cli\pump.py", line 504, in run
    rv_batch, batch = self.source.provide_batch()
  File "C:\Source\cb-cli\couchbase-cli\pump_dcp.py", line 254, in provide_batch
    rv, batch = self.provide_dcp_batch_actual()
  File "C:\Source\cb-cli\couchbase-cli\pump_dcp.py", line 413, in provide_dcp_batch_actual
    msg = (cmd, vbucket_id, key, flg, exp, cas, bytes([rev_seqno]), val, seqno, dtype, metalen,
ValueError: bytes must be in range(0, 256)
Is anyone looking into this?

With regards to error you 're seeing I will investigate that further.

pvarley · April 16, 2019, 11:22am

With regards to error you 're seeing I will investigate that further.

There was a bug introduce when moving to Python3 where cbbackup was not correctly converting seqno_rev from a int to bytes. Seqno_rev is 4 bytes long and Python3 bytes([Seqno_rev]) will convert to one byte.

Thank you for reporting this issue, I have opened a defect for it - MB-33810.

lukeb · April 17, 2019, 1:18am

Thanks for your advice, I’ll pull the 5.1.3 release.

Out of interest why are you downloading from GitHub instead of using one of the prebuild package?

I’m running the backup tool from a separate docker instance. I only need the cbbackup script so I didn’t want to install the entire Couchbase server. Is there another source I can get the prebuilt backup script from or is it only available as part of Couchbase installation?

pvarley · April 17, 2019, 9:23am

I have fixed the bug that was being hit.

Is there another source I can get the prebuilt backup script from or is it only available as part of Couchbase installation?

Unfortunately there is no another source, however as I mention previously if the release tag is used on the git repo then that should be more stable than using the head.

lukeb · April 18, 2019, 1:55am

Yep, I changed my dockerfile to checkout that specific commit and it’s now working. Thanks for your help.

Prachi_Katti · March 19, 2021, 12:09am

This worked for me.