Unable to use python scripts from lib in functions

Question

Here is an example that is structured much the same way as my project, this follows all the standards as outlined in the documentation.

.
├── lib
│   └── shared
│       └── common.py
├── packages
│   └── my-namespace
│       ├── my-function
│       │   ├── .include
│       │   ├── __main__.py
│       │   ├── build.sh
│       │   └── requirements.txt

packages/my-namespace/my-function/.include

../../../lib/shared
__main__.py

packages/my-namespace/my-function/build.sh

#!/bin/bash
set -e
virtualenv --without-pip virtualenv
pip install -r requirements.txt --target virtualenv/lib/python3.11/site-packages --compile

packages/my-namespace/my-function/__main__.py

from shared.common import Common

def main(event, context):
    return Common.foo()

packages/lib/shared/common.py

import os

class Common:
    def foo(self):
        return {
            'statusCode': 200,
            'headers': {
                'Content-Type': 'application/json'
            },
            'body': {
                'cwd': os.getcwd()
            }
        }

However, when calling the function it simply fails with the generic error message:

{
    "code": "69c77f0d4dca6d6f2a9a79ad065f2212",
    "error": "There was an error processing your request."
}

I have identified that the problem occurs when trying to import the python script that is included from lib. I have ensured: that ./shared/common.py exists within the functions directory on deployment and, that the code in ./shared/common.py is free from errors and that the scripts are executable.

If I forgoe the use of lib and instead copy shared directly into the my-function folder (effectively manually performing the include step, resulting in an identical folder structure at runtime), then the import works without issue and the code runs correctly, without any other changes required.

This defies logic and I can only conclude that there is some issue with the build process on DigitalOcean’s end that is resulting in scripts copied by the build process being inaccessible to python at runtime.

Speculatively, perhaps by way of some sort of pre-compilation/caching process rendering them as invisible/invalid to the runtime.

Whatever the cause, it renders it impossible to use lib functionality in any meaningful way.

GamesmenJordan · Answer

Upon further testing it appears the issue is that when using .include the build.sh script doesn’t appear to run, meaning no virtualenv is created and import fails for any not built-in packages meaning my example is incorrect.

Instead modifying shared/common.py to the following:

import os
import numpy

class Common:
    def foo(self):
        return {
            'statusCode': 200,
            'headers': {
                'Content-Type': 'application/json'
            },
            'body': {
                'cwd': os.getcwd()
            }
        }

And adding numpy to the requirement.txt file would demonstrate the issue. Now from shared.common import Common would fail with the error No module named 'numpy' when using .include - but it would work correctly if no .include file is used and the shared folder is manually copied.

Bobby Iliev · Answer

Hey Jordan! 👋

Is your function by any chance open-source? If so can you share a link to your GitHub repository so I could look into this further?

I think that it is possible that you might need a build.sh file within the lib directory in order for this to work as expected. Can you confirm if you have that in place already?

As far as I can see in the docs here, to include the lib/shared/common.py in your function’s deployment, you need to explicitly reference it in the .include file. Update the .include file in packages/my-namespace/my-function/ to:

../../../lib/shared

This tells the build process to include the shared directory in the deployment package.

Also, your build.sh looks fine, but make sure it has execute permissions which I’ve seen cause issues in some cases. You can set the execute permission using: