Functions: Cannot import a module inside a virtual environment

Question

Hello,

When using DO Functions, I require to import a Python module and once I do, the Function stops working with an The function exhausted its memory and was aborted. error.

When using doctl, the following message appears: Request accepted, but processing not completed yet. All functions invocation >= 30s will get demoted to an asynchronous invocation.

My example function:

from tronpy import Tron


def main(args):
    client = Tron()
    wallet = client.generate_address()
    return {
        "body": {
            "wallet": {
                "base58check_address": wallet["base58check_address"],
                "hex_address": wallet["hex_address"],
                "private_key": wallet["private_key"],
                "public_key": wallet["public_key"],
            },
        },
    }

This code is valid when run separately in my IDE but doesn’t work when using DO Functions.

I confirm the execution stops at from tronpy import Tron because I also tested this by removing further library function calls.

I also tried this with just the mysql-connector-python library, and it also doesn’t work, so that rules out tronpy failing.

My build.sh file:

#!/bin/bash

# Stop on the first error found
set -e

# Clean build and virtual environment
rm -rf venv
rm -f __deployer__.zip

# Create virtual environment and install dependencies
python3.11 -m venv --without-pip venv
source venv/bin/activate
# --compile --no-cache-dir
pip install "tronpy==0.4.0" --target venv/lib/python3.11/site-packages
deactivate

Notes about my build.sh file:

if the directory was virtualenv as docs say, the function doesn’t even run with standard library code.
I’m not using -r requirements.txt, but the result is equivalent if I did that.

My .include file just in case:

venv
__main__.py

Thanks in advance,

Ivan Montilla Miralles · Accepted Answer

I found out a workaround (but it’s not the proper, real solution), I proceed to explain:

In your Function source directory, you may have the venv directory, and it’s not included in Python’s sys.path; therefore, Python cannot see it to import (and running it, will fail).

Failure version

import sys

print(sys.path)
#  result: ["/lib", "/usr/local/lib/python311.zip", "/usr/local/lib/python3.11", "/usr/local/lib/python3.11/lib-dynload", "/usr/local/lib/python3.11/site-packages", "/tmp"]
# venv is not in this list!!

# This will fail, and return a resources exhaustion error (memory or timeout)
from tronpy import Tron

Workaround that works for me

import sys

# Append virtual environment directories
sys.path.append("venv")
sys.path.append("venv/lib/python3.11/site-packages")

# Any library in your virtual environment will now work correctly, in my example is TronPy
from tronpy import Tron

Real fix: Additional scenario with `virtualenv`

build.sh

set -e

rm -rf virtualenv
rm -f __deployer__.zip

virtualenv --without-pip virtualenv
pip install "tronpy==0.4.0" --target virtualenv/lib/python3.11/site-packages

In this scenario you don’t need to append anything to sys.path as DigitalOcean does it for you:

import sys

print(sys.path)
# result: ["/tmp/virtualenv/lib/python3.11/site-packages", "/lib", "/usr/local/lib/python311.zip", "/usr/local/lib/python3.11", "/usr/local/lib/python3.11/lib-dynload", "/usr/local/lib/python3.11/site-packages", "/tmp"]

However, be careful with local builds as Ubuntu 22.04.3 LTS ships with Python 3.10.12 and therefore virtualenv will result in the wrong Python version and your Python Function will not run, at all. Using --remote-build fixes this, until you can use the appropriate Python version locally.

Lesson learned: DigitalOcean will append the directory /tmp into sys.path, but that’s also the home of /tmp/virtualenv... which if the directory is not called exactly virtualenv, it will not work.

The docs mention it should be called virtualenv, but the remote builder logs will not tell you it failed to append virtualenv because it doesn’t exist. 8 hours hunting the root cause, in case you were wondering.

It also doesn’t help that the failure to import a library, appears as memory exhaustion error.