Oracle to Couchbase

@vsr1 - CURL access is unrestricted for me in query workbench so I CURL to any websites ?

@binh.le - Dear Binh , Any luck resolving my problem . Let me if there is any log which can help you to find whats going wrong ? Also we can we do massive data migration (in billions of records) using CURL approach as I think CURL has some limitations of 24 MB data ? And if this reasonable approach .

thanks

@binh.le - Dear Binh , I have managed to pull the data to CB bucket now as I had typo … Can you please help the right approach to pull the data pragmatically ? we have millions of records and CURL has limitation so we want to make the approach correct . thanks for your help !

Hi Eldorado,

You could use Oracle export to CSV and Couchbase cbimport tool to perform the data migration. However, if you are looking to migrate the Oracle data into Couchbase programmatically, then you could consider the followings:

  1. To use Couchbase Python SDK. https://docs.couchbase.com/python-sdk/current/start-using-sdk.html. The script can be written to call and execute the N1QL Query provided in my Oracle to Couchbase blog.
  • What you do need to do is to add the Oracle clause “OFFSET … FETCH” to the Oracle SQL statement.
  • You will also need to create a FOR loop which you will then incrementally pass in the OFFSET value in each iteration.
  • The FETCH NEXT n rows value would depend on the size of your data records. i.e. OFFSET 1 FETCH NEXT 1000. Then OFFSET 1001 FETCH NEXT 1001, and so on.
  • The Oracle query should have an ORDER BY to ensure that you don’t miss any records in the subsequent calls.
    Note: This approach is sufficient if you are only migrating a complete Oracle table to Couchbase. However if you also need to transform the Oracle data and MERGE the result into Couchbase array of document, then you may want to consider the second option.
  1. The second option would be to use Oracle OCI with Python. https://blogs.oracle.com/oraclemagazine/perform-basic-crud-operations-with-cx-oracle-part-2. Here you will submit the SQL query to Oracle then fetch the result one at a time. You then use the Couchbase Python SDK to perform the INSERT.

-binh

thanks Binh .
I think 2nd option what we want to opt for . CURL option look s like not better for us .
thanks for the suggestion . May be a future suggestion for CB to make the life easier for the data load from Oracle or any other relational DB to CB bucket .