Bigger log.00000000000000.data file created inside mainIndex

Hi,

I am trying to copy couchbase data (directory - /opt/couchbase/var/lib/couchbase/data) from linux to window through winscp.

But the size of log file log.00000000000000.data under index directory is too big to copy.

/opt/couchbase/var/lib/couchbase/data/@2i/books_#primary_14723171795409150663_0.index/mainIndex/log.00000000000000.data

why this file is too big although i have only 20 records in books bucket.

My couchbase version is - 6.0.1-2037

Thanks,
Inderjeet Singh

@inder, could you please tell me the size of the log data file and the frequency with which the 20 records are being updated?

Couchbase currently does not support moving index data files across nodes. If you are looking to replace your linux node with a windows node, you should consider swap rebalance:
https://docs.couchbase.com/server/6.0/install/upgrade-online.html#swap-rebalance-example-with-an-extra-node-available
Doing this will trigger rebuild of all indexes that were on the removed node on the new node. Please let me know if there is some other reason for copying the files.

@akhilmd

Thanks for your reply.

The size of the log data file is 56 Mb and these 20 records are getting updated 2-3 times every day from last 3-4 months.

My requirement is to copy UAT Environment data (Linux Server)to my local machine (Windows) for issue reproduction.
It’s not related with node swapping.

I also tried data export using cbexport and cbbackupmgr but these tool are not able to connect and copy data from Linux server to my local machine.

is there any better way i can use ?

Are you sure that the index file is that big? Couchbase uses sparse files. On my system it’s f.e. :

ls -lh log.00000000000002.data
-rwxr-x— 1 couchbase couchbase 3.8G Jan 23 08:34 log.00000000000002.data

du -h log.00000000000002.data
89M log.00000000000002.data

If you try to copy this file to another system it will copy the full 3.8G instead of the real used 89M

@gizmo74

You are right

After doing - ls -lh log.00000000000000.data
-rwxr-x—. 1 couchbase couchbase 1.3G Jan 23 09:20 log.00000000000000.data

it is showing me 1.3 G and copying the full 1.3 G on my local machine .

@inder, cbbackupmgr is the way to go here if you want to build the same indexes again on windows. What is the exact error that you saw when trying to use cbbackupmgr?

I am able to backup from one machine to my local machine by following instructions at https://docs.couchbase.com/server/current/backup-restore/cbbackupmgr-tutorial.html

On windows:

  1. Configure a backup directory
  2. Create a backup locally of the cluster by specifying address of the linux machine and the correct port
  3. Setup cluster (on windows machine)
  4. Restore using the created backup

@akhilmd

I have tried as per your suggetion. But i am getting below error

C:\Program Files\Couchbase\Server\bin>cbbackupmgr backup -a C:/data/local -r devcluster -c <linux_server>:8091 -u Admin -p admin123

Backing up to 2020-01-24T12_31_40.4835966+05_30
Copying at 0B (estimating time remaining) 0 items / 0B
books [ ] 100.00%
Error backing up cluster: dial tcp 13.93.72.143:8094: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

Even checking this folder - 2020-01-24T12_31_40.4835966+05_30, I can see folder for each bucket.
But i am not sure whether export is happened correctly because of the above error.

Below is the output for -
C:\Program Files\Couchbase\Server\bin>cbbackupmgr list -a C:/data/local
Size Items Name
291.14KB - /
291.14KB - + devcluster
291.14KB - + 2020-01-24T12_37_49.2115594+05_30
28.00KB - + books
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + chats
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + email
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + locale
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + misc
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + news
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + qna
28.00KB 0 + data
28.00KB 0 shard_0.fdb
39.14KB - + staging
411B 0 bucket-config.json
28.00KB 0 + data
28.00KB 0 shard_0.fdb
10.74KB 7 gsi.json
2B 0 views.json
28.00KB - + tutorials
28.00KB 0 + data
28.00KB 0 shard_0.fdb
28.00KB - + user
28.00KB 0 + data
28.00KB 0 shard_0.fdb

Hi @inder,

We can see the process crashed while trying to communicate with the Full Text Search Service to backup the Full Text Index definitions. Based on the error, the reason why I think this happening is that cbbackupmgr is trying to communicate using the internal IP for the Full Text Search Service. The version of cbbackupmgr included with 6.0.x doesn’t support alternative address (support was added in 6.5.0).

There are two options here:

  1. Use the config command to disable backing up Full Text Search. Note however, that this issue may arise when attempting to backup the Index Service.
  2. If this is for non production use, attempt the backup again using cbbackupmgr from 6.5.0.

Without any system collection logs we can’t guarantee that this is the exact issue you are experiencing but it’s likely that this is the issue.