Cbbackup/cbrestore trouble

steeve.descarpentrie · August 30, 2018, 7:29am

Hi,

I have a bucket (mobilotis) in a couchbase server 5.0.1 community edition, on an Windows 2016 server, the web console display 41214 items.
I have a unique index :

CREATE INDEX TableI ON mobilotis(table)`

When I’m running this query select count(*) from mobilotis where table = “commune”; the result is 38933.

Ok, I’m trying to duplicate the base for developpment purpose, so i’m using cbbackup like that :

C:\Program Files\Couchbase\Server\bin>cbbackup.exe http://localhost:8091 “C:\Quadra Diffusion\Sauvegardes\Couchbase” -m full -b mobilotis -u Administrator -p SecretPassword

I 7zipped the created backup folder et copied it in my computer, and un7ziped it of course.
I’m under wondows 10 last version, and I installed the exact same server version, and I added the same bucket, empty of course.

I’m using the cbrestore like that :

C:\Program Files\Couchbase\Server\bin>cbrestore.exe c:\Dump\Couchbase\ http://localhost:8091 -b mobilotis -B mobilotis -u Administrator -p SecretPassword

This command goes without trouble.
After that, I check in the web console, on my computer, I have 41154 items instead of 41214 .
When I’m running this query select count(*) from mobilotis where table = “commune”; the result is 709 instead of 38933.

What am I doing wrong ?

Regards

Steeve

steeve.descarpentrie · September 3, 2018, 6:15am

Hi,

Who can share his experience with cbbackup/cbrestore ?

Steeve

steeve.descarpentrie · September 4, 2018, 8:08am

Hi,

I tried something otherwise :
Cbbackup of my bucket + Flush my bucket + cbrestore of my bucket.
I did it like that

cbbackup.exe http://localhost:8091 C:\Dump\Couchbase -b mobilotis -u Administrator -p MyPassword

Flush using the web app

cbrestore.exe c:\Dump\Couchbase\2018-09-04T074459Z\2018-09-04T074459Z-full http://localhost:8091 -b mobilotis -B mobilotis -u Administrator -p MyPassword

I had 41059 items before the backup, and 41060 items after the backup.
The result of the query should have told me around 39000 items, but it gave 927 items :

select count(*) from mobilotis where table = “commune”;

Who can explain that to me ?

Regards

Steeve

thuan · September 4, 2018, 7:29pm

Hi Steeve,
I follow your steps to reproduce your issue but I could not reproduce it in my system.

I loaded 100K documents to Couchbase Server Community 5.0.1 to bucket name bucket0 on Windows Server 2012 R2 64-bit.
UI showed 100K documents
I created index age in this bucket0
I backed up this bucek0 using cbbackup in bin directory

Blockquote
C:\Program Files\Couchbase\Server\bin>.\cbbackup.exe http://172.23.109.81:8091 c:\tmp\backup -u Administrator -p password -m full
[####################] 100.0% (100000/estimated 100000 msgs)
bucket: bucket0, msgs transferred…
: total | last | per sec
byte : 6968860 | 6968860 | 61354.8
done
C:\Program Files\Couchbase\Server\bin>

This backup execution generate folder name 2018-09-04T180935Z
Then I used 7 zip to zip backup folder in c:\tmp\backup to 2018-09-04T180935Z.zip
I install Couchbase Server Community 5.0.1 on a Windows 10 Pro 64-bit with build 1803 with data, index and query services (same as in Windows 2012 R2)
Copy zip file 2018-09-04T180935Z.zip from Windows 2012 R2 to Windows 10 Pro at same location c:\tmp\backup
Unzip the zip file and move zip file 2018-09-04T180935Z.zip out of backup folder.
Create an empty bucket0 on Windows 10 Pro
Run cbrestore to restore data back to bucket0 on Windows 10 Pro

Blockquote
C:\Program Files\Couchbase\Server\bin>.\cbrestore.exe c:\tmp\backup\2018-09-04T180935Z http://12.11.10.145:8091 -u Administrator -p password --bucket-source=bucket0 --bucket-destination=bucket0
[####################] 100.0% (100000/estimated 100000 msgs)
bucket: bucket0, msgs transferred…
: total | last | per sec
byte : 6968860 | 6968860 | 355753.7
done
C:\Program Files\Couchbase\Server\bin>

Check UI on restore server, I see 100K
Check indexes and I see 2 indexes definition restored back.
Build index on these 2 indexes. I could query all docs I want.

I will try again on Windows Server 2016 and update here soon.

thuan · September 4, 2018, 9:39pm

I tried the same steps above in my backup restore data with diffeent operating system. Backup data from bucket0 with index in Windows 2016 Standard and restore back to Windows 10 Pro, all data restore back to bucket0 successfully and queries are ok to get data.

Blockquote C:\Program Files\Couchbase\Server\bin>.\cbrestore.exe c:\tmp\backup\2018-09-04T211039Z http://12.11.10.145:8091 -u Administrator -p password --bucket-source=bucket0 --bucket-destination=bucket0
[####################] 100.0% (100000/estimated 100000 msgs)
bucket: bucket0, msgs transferred…
: total | last | per sec
byte : 6968860 | 6968860 | 331408.6
done
C:\Program Files\Couchbase\Server\bin>

It might relate to your environment or something else. If it is possible, could you share with us your data so I could try it again in our environment?
Thanks

steeve.descarpentrie · September 5, 2018, 5:35am

Hi Thuan,

Thank you very much for this feedback.

Can Sync gateway have an impact ?

How can I send you my backup ?

Steeve

steeve.descarpentrie · September 5, 2018, 6:26am

Hi again,
I tried to mimic your command line, I have this strange error message :

C:\Program Files\Couchbase\Server\bin>cbbackup.exe http://localhost:8091 c:\tempJohnson -u Administrator -p MyPassword -m full
.
bucket: controle, msgs transferred…
: total | last | per sec
byte : 0 | 0 | 0.0
[####################] 99.9% (41375/estimated 41435 msgs)
bucket: mobilotis, msgs transferred…
: total | last | per sec
byte : 31461194 | 31461194 | 1172962.3
done

C:\Program Files\Couchbase\Server\bin>cbrestore.exe c:\TempJohnson\2018-09-05T061827Z http://localhost:8091 -u Administrator -p MyPassword–bucket-source=mobilotis --bucket-destination=controle
error: no valid backup directory found: c:\TempJohnson\2018-09-05T061827Z\2018-09-05T061827Z-full

Regards

Steeve

steeve.descarpentrie · September 5, 2018, 7:26am

HI,

I finaly change the command line like that to make it work :

cbrestore.exe c:\TempJohnson\2018-09-05T061827Z\2018-09-05T061827Z-full http://localhost:8091 -u Administrator -p MyPassword --bucket-source=mobilotis --bucket-destination=controle

controle is another bucket in the same server than mobilotis. Some interesting infos :
controle 41163 items, Disk used 48.9 Mb
mobilotis 41223 items, Disk used 101 Mb
Note that I have the same result when I use the query : select count(*) from controle;

I created this index on the 2 buckets :

CREATE INDEX IdI ON controle((meta().id))

Here begin the very interesting count, this query give me the same result on the 2 buckets :

select count(*) from controle (or mobilotis) WHERE META().id NOT LIKE “_sync%”;
40116

idem, for the query :

select count(*) from controle (or mobilotis) WHERE META().id LIKE “commune%”;
38933

I finally test with a lower count to be able to display all the informations :

select count(*) from controle (or mobilotis) WHERE META().id LIKE “reservation%” ;
32
So I did the query :

select * from mobilotis WHERE META().id LIKE “reservation%” ;

I have a good result :

[
{
“mobilotis”: {
“dateCreation”: “2018-07-30T12:33:55.2527826+02:00”,
“dateEncaissementAcompte”: “2018-08-23T00:00:00.0000000+02:00”,
“dateModif”: “2018-08-01T11:49:20.2596280+02:00”,
“dateSignaturePv”: “2018-07-23T00:00:00.0000000+02:00”,
“droitDeMutation”: false,
“financements”: ,
“futurAcquereur”: {
“adresse”: {
“codePostal”: “94140”,
…

But a strange one with the bucket controle

select * from controle WHERE META().id LIKE “reservation%” ;

[
{
“controle”: “<binary (3800 b)>”
},
{
“controle”: “<binary (3463 b)>”
},
{
“controle”: “<binary (4421 b)>”
},

Now I understand why my index is not working on the field “table”, but Why the content of the document is like that ?? What is this binary translation ?

Regards

Steeve

vsr1 · September 5, 2018, 5:53pm

It says the data is not in JOSN format. It saying non-josn ( size bytes) N1QL works on JOSN documents only.

select META().id, * from controle WHERE META().id LIKE “reservation%” ;
Goto UI and search document and check it out or paste the document here.

hod.greeley · September 5, 2018, 8:07pm

Just a quick note. Sync Gateway can add a small number of extra items to a bucket. That should be exported in a backup and show on import, though.

steeve.descarpentrie · September 6, 2018, 5:39am

Hi,
I did that yesterday, the document is empty.

How a json document, backuped with cbbackup, can be transform in a non json document after restoring with cbrestore ?
What in my command line did I do wrong ?

Steeve

vsr1 · September 6, 2018, 12:22pm

@anil could you please answer this.

steeve.descarpentrie · September 10, 2018, 6:55am

Hi,

I really need your help guys, I need to restore this backup to test my dev with real datas.

Thank you

Steeve

pvarley · September 10, 2018, 9:46pm

Hello @steeve.descarpentrie,

You are hitting a known issue with cbbackupwhere documents can get corrupted if extended Attributes(Xattr) are used. Newer versions of Couchbase Sync Gateway do use Xattrs.

This issue is fixed in Couchbase Server CE versions 5.1.0. cbbackup from Couchbase Server 5.1.X can be used against Couchbase Server 5.0.1.

steeve.descarpentrie · September 11, 2018, 6:29am

Hi, Thank you very much, I will upgrade and test it right now.

Steeve

steeve.descarpentrie · September 11, 2018, 8:36am

I won’t be able to backup my bucket before upgrading the server, which is not cool, cause my first test to upgrade isn’t positif, I opened a topic here

May I use the cbbackup/cbrestore of a 511’s version with a 501’s server ?

Regards
Steeve

pvarley · September 11, 2018, 9:19am

Yes, use the cbbackup from 5.1.1 it will work with Couchbase Server 5.0.1.

Afsal_kritisoft · May 5, 2021, 11:26am

I am facing similar issue ?

Any update on how the issue is resolved ?