Mongodb to couchbase


#1

What step will be needed to transfer about a 1.2gb database from mongodb to couchbase.

This 1.2gb of data has several collections and each collection is in its own json file.

please list the steps.

thank you

Michelle


#2

Hello,

As you may know, inside Couchbase we are a real “schema-less” database in the sense that you can put all your different JSON (collections) into a single bucket – if this is compatible with your application logic. So let’s say you have your application bucket created into Couchbase.

It is also important to choose the key, you will probably use the object_id attribute from your collection items.

Now you have different approaches to “move the data”.

1- Programmatic approach:
You use a program -what is your favorite language?- and connect to Mongo and Couchbase and iterate on each collection, and serialize the DBObject into a JSON document into Couchbase. I have done a small tool for SQL available here: https://github.com/tgrall/couchbase-sql-importer , You can look at it to see how it works and extend it for Mongo. -I will look at that later this week-

2- Use an ETL Tool (Extract Transform and Load)
Couchbase and Mongo have connectors for Talend so you can use a tool like that to connect the 2 systems and do some transformation.

3- Couchbase Document Loader cbdocloader
In this case you need to “export” your Mongo data in the good format for Couchbase, that is quite simple:

  • file name will be the key of the document
  • file content should be the JSON document itself

Let me know if you need more information.

Regards
Tug
@tgrall


#3

I think this would make an excellent blog post or how-to section on the site (if it doesn’t already exist).


#4

Tug

We have tried both export types in mongodb ( json and jsonarray) neither format worked with cbdocloader. but both have errors.

any suggestions.

Michelle


#5

Hello,

You cannot use the cbdocloader directly on a Mongo export file.

The Mongo file contains either:

  • one json document by line
  • an array of json documents

where cbdocloader tool is using 1 file for each json document.

Also Mongo is using special types for ids and date, so it is sometimes useful to serialize it in a proper JSON format.

I have created a smal tool that allows you to copy your Mongo database/collections directly into Couchbase or you can export these documents into a directory that you can use directly with cbdocloader tool.

You can find the developer version here:

Note: feel free to test and give me feedback (even if this is still early stage).

Regards
Tug
@tgrall


#6

Good idea Brad, let’s work on this migration first and document it :wink: