Error 'Failed to send key" when running cbimport

Hi,

I am running Couchbase Server community edition v6.6.0 and I am trying to use cbimport to do a data load. Here is an example of the file I am trying to load:

[
{
“fleetid”:“300”,
“name”:“5.5 Metre”,
“type”:“fleet”
},
{
“fleetid”:“600”,
“name”:“Hansa 2.3”,
“type”:“fleet”
},
{
“fleetid”:“150”,
“name”:“J/22”,
“type”:“fleet”
}
]

When looking at my bucket, I have no document with a key of fleet::300 and yet, when I run the following cbimport command, it fails: cbimport json --cluster couchbase://server1 --bucket bucket1 --dataset file:///home/data/fleet.json --errors-log ./fleet.json.log --format list --username XXXXX --password XXXXX --verbose --generate-key fleet::%fleetid%
with the following error message:

2021-05-26T16:01:31.238-06:00 (Rest) GET http://server1:8091/pools/default 200
2021-05-26T16:01:31.238-06:00 (Plan) Transferring Eventing metadata
2021-05-26T16:01:31.238-06:00 (Plan) Successfully transferred Eventing metadata
2021-05-26T16:01:31.238-06:00 (Plan) Transferring full text index aliases
2021-05-26T16:01:31.238-06:00 (Plan) Successfully transferred full text index aliases
2021-05-26T16:01:31.238-06:00 (Plan) Transferring bucket configuration for ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.238-06:00 Unable to restore bucket settings, none exist
2021-05-26T16:01:31.238-06:00 (Plan) Successfully transferred bucket configuration for ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.238-06:00 (Plan) Transferring Analytics metadata
2021-05-26T16:01:31.238-06:00 (Rest) cannot restore empty analytics metadata
2021-05-26T16:01:31.238-06:00 (Plan) Successfully transferred Analytics metadata
2021-05-26T16:01:31.238-06:00 (Plan) Transferring full text index definitions for ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.238-06:00 (Plan) Successfully transferred full text index definitions for ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.238-06:00 (Plan) Transferring views definitions from ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.298-06:00 (Rest) GET http://server1:8091/pools/default/nodeServices 200
2021-05-26T16:01:31.298-06:00 (Plan) Successfully transferred views definitions from ‘bucket1’ to ‘bucket1’
2021-05-26T16:01:31.298-06:00 (Plan) Executing queries against ‘bucket1’ from ‘bucket1’
2021-05-26T16:01:31.361-06:00 (Rest) GET http://server1:8091/pools/default/nodeServices 200
2021-05-26T16:01:31.362-06:00 (Plan) Successfully executed queries against ‘bucket1’ from ‘bucket1’
2021-05-26T16:01:31.362-06:00 (Plan) Checking for data movement restrictions between bucket1 and bucket1
2021-05-26T16:01:31.362-06:00 Transferring to Couchbase Server 6.6.0
2021-05-26T16:01:31.362-06:00 (Plan) Successfully checked for data movement restrictions between bucket1 and bucket1
2021-05-26T16:01:31.362-06:00 (Plan) Deciding which key value data to transfer for bucket1
2021-05-26T16:01:31.362-06:00 (Plan) Successfully decided which key value data to transfer for bucket1
2021-05-26T16:01:31.362-06:00 (Plan) Transferring new key value data for bucket1 to bucket1
2021-05-26T16:01:31.462-06:00 (Rest) GET http://server1:8091/pools/default/buckets 200
2021-05-26T16:01:31.697-06:00 (Rest) GET http://server1:8091/pools/default/nodeServices 200
2021-05-26T16:01:53.362-06:00 (Pool) (bucket1) Failed to send key ‘fleet::300’ operation timed out, will retry
2021-05-26T16:02:23.463-06:00 (Pool) (bucket1) Failed to send key ‘fleet::300’ operation timed out, will retry
2021-05-26T16:02:33.362-06:00 WARN: (Pool) (bucket1) Memcached has been inactive for 1m0s, last item count 0 – couchbase.(*MemcachedWorker).monito
rActivity.func1() at pool_worker.go:411
2021-05-26T16:03:03.613-06:00 (Pool) (bucket1) Failed to send key ‘fleet::300’ operation timed out, will retry
2021-05-26T16:03:53.813-06:00 (Pool) (bucket1) Failed to send key ‘fleet::300’ operation timed out, will retry
2021-05-26T16:04:54.064-06:00 (Pool) (bucket1) Failed to send key ‘fleet::300’ operation timed out, will retry
2021-05-26T16:04:54.364-06:00 WARN: (Pool) (bucket1) Failed to send key ‘fleet::300’ because we have reached the max retries – couchbase.
(*MemcachedWorker).dispatchOperation() at pool_worker.go:255
2021-05-26T16:04:54.364-06:00 JSON import failed: 0 documents were imported, 1 documents failed to be imported
JSON import failed: 0 documents were imported, 1 documents failed to be imported
2021-05-26T16:04:54.364-06:00 JSON import failed: failed to execute cluster operations: failed to execute bucket operations: failed to transfer buck
et data for bucket ‘bucket1’: failed to transfer key value data: failed to transfer key value data: operation failed: max retries reached
JSON import failed: operation failed: max retries reached

Any idea what is happening? The import seems to be stuck on that particular line and doesn’t move forward, but that line has nothing different from anything else, so I am at a loss here. I have tried to look at the logs on the server side and nothing jumps out.

Cheers in advance for the help,
Bertrand.

Hi Bertrand,
Thanks for using our product. I have a question. What is your operating system which Couchbase server is running on?
In mean time, I will try to reproduce this issue in CentOS 7.6.
Thanks
Thuan

Hi Bertrand,
I did try on my Centos 7.6 server with your sample json file, it works as expected.

[root@s44015 ~]# /opt/couchbase/bin/cbimport json --cluster localhost --bucket bucket1 --dataset file:///root/fleet.json --format list --username Administrator --password password --verbose --generate-key fleet::%fleetid%
2021-06-03T11:44:50.932-07:00 (Rest) GET http://localhost:8091/pools/default 200
2021-06-03T11:44:50.933-07:00 (Plan) Transferring Eventing metadata
2021-06-03T11:44:50.933-07:00 (Plan) Successfully transferred Eventing metadata
2021-06-03T11:44:50.933-07:00 (Plan) Transferring full text index aliases
2021-06-03T11:44:50.933-07:00 (Plan) Successfully transferred full text index aliases
2021-06-03T11:44:50.933-07:00 (Plan) Transferring bucket configuration for ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.933-07:00 Unable to restore bucket settings, none exist
2021-06-03T11:44:50.933-07:00 (Plan) Successfully transferred bucket configuration for ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.933-07:00 (Plan) Transferring Analytics metadata
2021-06-03T11:44:50.933-07:00 (Rest) cannot restore empty analytics metadata
2021-06-03T11:44:50.933-07:00 (Plan) Successfully transferred Analytics metadata
2021-06-03T11:44:50.933-07:00 (Plan) Transferring full text index definitions for ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.933-07:00 (Plan) Successfully transferred full text index definitions for ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.933-07:00 (Plan) Transferring views definitions from ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.936-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
2021-06-03T11:44:50.937-07:00 (Plan) Successfully transferred views definitions from ‘bucket1’ to ‘bucket1’
2021-06-03T11:44:50.937-07:00 (Plan) Executing queries against ‘bucket1’ from ‘bucket1’
2021-06-03T11:44:50.938-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
2021-06-03T11:44:50.938-07:00 (Plan) Successfully executed queries against ‘bucket1’ from ‘bucket1’
2021-06-03T11:44:50.939-07:00 (Plan) Checking for data movement restrictions between bucket1 and bucket1
2021-06-03T11:44:50.939-07:00 Transferring to Couchbase Server 6.6.0
2021-06-03T11:44:50.939-07:00 (Plan) Successfully checked for data movement restrictions between bucket1 and bucket1
2021-06-03T11:44:50.939-07:00 (Plan) Deciding which key value data to transfer for bucket1
2021-06-03T11:44:50.939-07:00 (Plan) Successfully decided which key value data to transfer for bucket1
2021-06-03T11:44:50.939-07:00 (Plan) Transferring new key value data for bucket1 to bucket1
2021-06-03T11:44:50.949-07:00 (Rest) GET http://localhost:8091/pools/default/buckets 200
2021-06-03T11:44:50.953-07:00 (Rest) GET http://localhost:8091/pools/default/nodeServices 200
2021-06-03T11:44:51.022-07:00 (Plan) Data transfer completed after 2.748306ms
2021-06-03T11:44:51.023-07:00 (Plan) {“bytes_received”:355,“mutations_received”:3,“complete”:true,“errored”:false}
2021-06-03T11:44:51.023-07:00 (Plan) Transfer for cluster complete
2021-06-03T11:44:51.023-07:00 (Plan) Transfer of all data complete
JSON file:///root/fleet.json imported to http://localhost:8091 successfully
Documents imported: 3 Documents failed: 0
[root@s44015 ~]#
[root@s44015 ~]# more /opt/couchbase/etc/runtime.ini
[product]
title = Couchbase Server
version = 6.6.0-7909
license = community
[root@s44015 ~]#


You may check json format in you sample. Double quote is ", not ” as in your sample.
If you have any question, please drop me a line at thuan at couchbase dot com
Thanks
Thuan Nguyen

Hi Thuan,

thank you for looking into this. Couchbase is running on Ubuntu 20.04.2 LTS with kernel version 5.4.0-1038, because this is a test environment, the specs of this virtual machine are lower than recommended. It has 4GB of RAM and 2 vCPUs. Could this be the reason for the issue?
If it is, then why would it works for some records and not others?

Regarding the double quote, I have checked my file and they are the proper double quotes. This is just a formatting error in the forums. :slight_smile:

Thanks a lot,
Bertrand.