How to create AWS Lambda deployment package that uses Couchbase Python client

Python SDK
1

I’m trying to use AWS Lambda to transfer data from my S3 bucket to Couchbase server, and I’m writing in Python. So I need to import couchbase module in my Python script. Usually if there are external modules used in the script, I need to pip install those modules locally and zip the modules and script together, then upload to Lambda. But this doesn’t work this time. The reason is the Python client of couchbase works with the c client of couchbase: libcouchbase. So I’m not clear what I should do. When I simply add in the c client package (with that said, I have 6 package folders in my deployment package, the first 5 are the ones installed when I run “pip install couchbase”: couchbase, acouchbase, gcouchbase, txcouchbase, couchbase-2.1.0.dist-info; and the last one is the c client of Couchbase I installed: libcouchbase), lambda doesn’t work and said:

“Unable to import module ‘lambda_function’: libcouchbase.so.2: cannot open shared object file: No such file or directory”

Any idea on how I can get the this work? With a lot of thanks.

1 Like
2

I’m not familiar with AWS lambda, but if you haven’t actually installed libcouchbase you might also need to set LD_LIBRARY_PATH (i.e. the linker runtime path) to indicate to the kernel where to find libcouchbase.so. This will usually be in <C_SDK_INSTALL_PATH>/lib

1 Like
3

Hi, Did it get it to work? I’m about to attempt the same.

4

Has anyone been able to connect from AWS Lambda Python 3.6 to couchbase?

I get the same error as this,

“Unable to import module ‘lambda_function’: libcouchbase.so.2: cannot open shared object file: No such file or directory”

There is no way to ssh into aws lambda machines, so I has someone able to achieve to set the a library path <C_SDK_INSTALL_PATH>/lib

5

You have to install libcouchbase there: https://developer.couchbase.com/documentation/server/current/sdk/c/start-using-sdk.html

We have yum repositories and also you have install it directly from RPMs

2 Likes
6

Thank you for your link, I got this working.

For others who are searching to fix this error, run the following in your docker container running Amazon AMI (I used docker image dacut/amazon-linux-python-3.6, it should be the same in ec2 or any Amazon Linux)

Only needed during first-time setup:

wget http://packages.couchbase.com/releases/couchbase-release/couchbase-release-1.0-4-x86_64.rpm
sudo rpm -iv couchbase-release-1.0-4-x86_64.rpm

Will install or upgrade existing packages

sudo yum install libcouchbase-devel libcouchbase2-bin gcc gcc-c++

Then include the files in

/usr/include/libcouchbase

and the .so files in

/usr/lib64/libcouchbase.so
/usr/lib64/libcouchbase.so.2
/usr/lib64/libcouchbase.so.2.0.51

in your lambda deployment package zip file which you will be uploading to S3.

(if you cannot for some reason find the files in /usr/lib64, use this command to check where the files are located - bash-4.2#rpm -ql libcouchbase-devel libcouchbase2-bin )

1 Like
7

I can confirm that this still works as of January 2020

For those who need an example Dockerfile this is what I used to build via serverless framework:

FROM lambci/lambda:build-python3.7
RUN yum -y install wget
RUN wget http://packages.couchbase.com/releases/couchbase-release/couchbase-release-1.0-6-x86_64.rpm
RUN rpm -iv couchbase-release-1.0-6-x86_64.rpm
RUN yum -y update
RUN yum -y install libcouchbase-devel libcouchbase2-bin gcc gcc-c++
8

I’m actually facing the same problem, I want to deploy an AWS lambda function which uses couchbase python sdk. The problem I’m facing is
{
“errorMessage”: “Unable to import module ‘lambda_function’: No module named ‘couchbase._libcouchbase’”,
“errorType”: “Runtime.ImportModuleError”
}
I want to know where do I find the libcouchbase and how to include it with my zip file.
Right now, what I did is zipped contents of site-packages folder and uploaded it to was lambda.
Please help!!
Thanks,
Abdul Ghani.

9

@juarezeppy
RUN pip3 install couchbase --target="./" still fails after your steps.

Then include the files in
/usr/include/libcouchbase

and the .so files in
/usr/lib64/libcouchbase.so
/usr/lib64/libcouchbase.so.2
/usr/lib64/libcouchbase.so.2.0.51

Wont we need to copy these into the lambda package too.