When trying to make backups the process gets interrupted


#1

I am trying to test couchbase enterprise edition 5.5

In my tests i am trying to do a full backup but it seems something happens in that process.

This is what i get :

2018-10-07 20:17:55,073: w0 no response for 30 seconds while there 1024 active streams
[ ] 0.1% (37151/estimated 34631186 msgs)
bucket: logs, msgs transferred…
: total | last | per sec
byte : 7103643 | 7103643 | 214181.1
done

Has anyone had this problem before ? Or do you have any ideea what it means ?


#2

Hi @ciprian,

Are you using the command line tool to perform a backup? Could you show exactly what the command is that you’re typing? Do you get the same error on a smaller bucket?


#3

Hi, sorry for the late reply.

I am using the following command : ./cbbackup couchbase://127.0.0.1:8091 /backups/10 -u SomeUser -p SomePassword -m full

I did not try on a small backet. But i think the error will not appear. On the big bucket sometimes i even reach 40%

2018-10-30 19:24:43,594: w0 no response for 30 seconds while there 1024 active streams
[######### ] 43.9% (15692540/estimated 35754538 msgs)
bucket: logs, msgs transferred…
: total | last | per sec
byte : 3440284582 | 3440284582 | 2707410.7

Any help will be appreciated.


#4

Hi @ciprian,

Is it possible that you are running cbbackup multiple times at once? If so, there is an outstanding issue around that. https://issues.couchbase.com/browse/MB-18957

If not, then we’ll need to create a new ticket. Could you please try it on a small bucket just to confirm that this only occurs for a large bucket? Could you also tell me what operating system you are using (and version information), as well as how many nodes are in your Couchbase cluster?


#5

Hi @matthew.groves,

I am very sure that only one instance if cbbackup is running,

The small bucket looks ok, but i had less than 100.000 records. Should i try with more?

I am using Ubuntu 18 , 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

system         D3417-B1 (S26361-Kxxx-Vyyy)
/0                      bus            D3417-B1
/0/0                    memory         64KiB BIOS
/0/32                   memory         128KiB L1 cache
/0/33                   memory         128KiB L1 cache
/0/34                   memory         1MiB L2 cache
/0/35                   memory         8MiB L3 cache
/0/36                   processor      Intel(R) Xeon(R) CPU E3-1275 v5 @ 3.60GHz
/0/37                   memory         64GiB System Memory
/0/37/0                 memory         16GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/37/1                 memory         16GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/37/2                 memory         16GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/37/3                 memory         16GiB DIMM DDR4 Synchronous 2133 MHz (0.5 ns)
/0/100                  bridge         Skylake Host Bridge/DRAM Registers
/0/100/2                display        HD Graphics P530
/0/100/14               bus            Sunrise Point-H USB 3.0 xHCI Controller
/0/100/14/0  usb1       bus            xHCI Host Controller
/0/100/14/1  usb2       bus            xHCI Host Controller
/0/100/14.2             generic        Sunrise Point-H Thermal subsystem
/0/100/16               communication  Sunrise Point-H CSME HECI #1
/0/100/17               storage        Sunrise Point-H SATA controller [AHCI mode]
/0/100/1c               bridge         Sunrise Point-H PCI Express Root Port #5
/0/100/1c/0             storage        NVMe SSD Controller SM961/PM961
/0/100/1d               bridge         Sunrise Point-H PCI Express Root Port #9
/0/100/1d/0             storage        NVMe SSD Controller SM961/PM961
/0/100/1f               bridge         Sunrise Point-H LPC Controller
/0/100/1f.2             memory         Memory controller
/0/100/1f.4             bus            Sunrise Point-H SMBus
/0/100/1f.6  enp0s31f6  network        Ethernet Connection (2) I219-LM
/1                      power          To Be Filled By O.E.M.

I have just one node .


#6

Hello @ciprian,

The error message means that cbbackup did not receive any data from the cluster. Would it be possible to run the command again with -vvv to increase the verbosity of the logging. It would also be worth checking the server side logs. The memcached.log will show what is happening with the cbbackup connection.


#7

Hi @pvarley,

I did as you said and here are the logs :
memcached.zip (82.7 KB)

Thanks for the help


#8

Hey @pvarley , @matthew.groves any new ideeas about this? Or any other suggestions ?


#9

The same happened to me.

w0 no response for 30 seconds while there 1024 active streams.

Any advises or perhaps suggestions?


#10

@ciprian Can you try this on Couchbase Server 6 please. I suspect you might be hitting MB-31481 where in rare cases the server does not send the snapshot end marker.


#11

@pvarley I have already tried it. No joy. Still the same issue.


#12

hey guys, any new ideas on this one ?