Question

App Platform multiprocessing (Python)

Posted December 16, 2020 434 views
PythonDigitalOcean App Platform

I’ve managed to deploy my app onto the App platform but when it runs, it now gives me an errno 38 Function not implemented?
Looking through the tracestack, this looks related to using multiprocessing pools in my app. Is this not supported by the App platform?

edited by MattIPv4
1 comment
  • Full Traceback as below

    portfolio-tracker | 2020-12-16 13:04:30 Traceback (most recent call last):
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
    portfolio-tracker | 2020-12-16 13:04:30     self.handle_request(listener, req, client, addr)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 190, in handle_request
    portfolio-tracker | 2020-12-16 13:04:30     util.reraise(*sys.exc_info())
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/util.py", line 625, in reraise
    portfolio-tracker | 2020-12-16 13:04:30     raise value
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
    portfolio-tracker | 2020-12-16 13:04:30     respiter = self.wsgi(environ, resp.start_response)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    portfolio-tracker | 2020-12-16 13:04:30     return self.wsgi_app(environ, start_response)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    portfolio-tracker | 2020-12-16 13:04:30     response = self.handle_exception(e)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    portfolio-tracker | 2020-12-16 13:04:30     reraise(exc_type, exc_value, tb)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    portfolio-tracker | 2020-12-16 13:04:30     raise value
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    portfolio-tracker | 2020-12-16 13:04:30     response = self.full_dispatch_request()
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    portfolio-tracker | 2020-12-16 13:04:30     rv = self.handle_user_exception(e)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    portfolio-tracker | 2020-12-16 13:04:30     reraise(exc_type, exc_value, tb)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    portfolio-tracker | 2020-12-16 13:04:30     raise value
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    portfolio-tracker | 2020-12-16 13:04:30     rv = self.dispatch_request()
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    portfolio-tracker | 2020-12-16 13:04:30     return self.view_functions[rule.endpoint](**req.view_args)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    portfolio-tracker | 2020-12-16 13:04:30     return func(*args, **kwargs)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/routes.py", line 32, in index
    portfolio-tracker | 2020-12-16 13:04:30     return update_pf()
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    portfolio-tracker | 2020-12-16 13:04:30     return func(*args, **kwargs)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/routes.py", line 95, in update_pf
    portfolio-tracker | 2020-12-16 13:04:30     df = pf.info_date(as_at_date, hide_zero_pos=hide_zero, no_update=no_update)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/portfolio.py", line 114, in info_date
    portfolio-tracker | 2020-12-16 13:04:30     prices_df = self.curr_positions(
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/portfolio.py", line 392, in curr_positions
    portfolio-tracker | 2020-12-16 13:04:30     prices_df = data.get_price_data(
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/utils/data.py", line 75, in get_price_data
    portfolio-tracker | 2020-12-16 13:04:30     with ThreadPool(processes=10) as pool:
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    portfolio-tracker | 2020-12-16 13:04:30     Pool.__init__(self, processes, initializer, initargs)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    portfolio-tracker | 2020-12-16 13:04:30     self._change_notifier = self._ctx.SimpleQueue()
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    portfolio-tracker | 2020-12-16 13:04:30     return SimpleQueue(ctx=self.get_context())
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    portfolio-tracker | 2020-12-16 13:04:30     self._rlock = ctx.Lock()
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    portfolio-tracker | 2020-12-16 13:04:30     return Lock(ctx=self.get_context())
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    portfolio-tracker | 2020-12-16 13:04:30     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
    portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    portfolio-tracker | 2020-12-16 13:04:30     sl = self._semlock = _multiprocessing.SemLock(
    portfolio-tracker | 2020-12-16 13:04:30 OSError: [Errno 38] Function not implemented
    portfolio-tracker | 2020-12-16 15:31:19 [2020-12-16 04:31:19 +0000] [12] [ERROR] Socket error processing request.
    portfolio-tracker | 2020-12-16 15:31:19 Traceback (most recent call last):
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
    portfolio-tracker | 2020-12-16 15:31:19     self.handle_request(listener, req, client, addr)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 190, in handle_request
    portfolio-tracker | 2020-12-16 15:31:19     util.reraise(*sys.exc_info())
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/util.py", line 625, in reraise
    portfolio-tracker | 2020-12-16 15:31:19     raise value
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
    portfolio-tracker | 2020-12-16 15:31:19     respiter = self.wsgi(environ, resp.start_response)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    portfolio-tracker | 2020-12-16 15:31:19     return self.wsgi_app(environ, start_response)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    portfolio-tracker | 2020-12-16 15:31:19     response = self.handle_exception(e)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    portfolio-tracker | 2020-12-16 15:31:19     reraise(exc_type, exc_value, tb)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    portfolio-tracker | 2020-12-16 15:31:19     raise value
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    portfolio-tracker | 2020-12-16 15:31:19     response = self.full_dispatch_request()
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    portfolio-tracker | 2020-12-16 15:31:19     rv = self.handle_user_exception(e)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    portfolio-tracker | 2020-12-16 15:31:19     reraise(exc_type, exc_value, tb)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    portfolio-tracker | 2020-12-16 15:31:19     raise value
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    portfolio-tracker | 2020-12-16 15:31:19     rv = self.dispatch_request()
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    portfolio-tracker | 2020-12-16 15:31:19     return self.view_functions[rule.endpoint](**req.view_args)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    portfolio-tracker | 2020-12-16 15:31:19     return func(*args, **kwargs)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/routes.py", line 32, in index
    portfolio-tracker | 2020-12-16 15:31:19     return update_pf()
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
    portfolio-tracker | 2020-12-16 15:31:19     return func(*args, **kwargs)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/routes.py", line 95, in update_pf
    portfolio-tracker | 2020-12-16 15:31:19     df = pf.info_date(as_at_date, hide_zero_pos=hide_zero, no_update=no_update)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/portfolio.py", line 114, in info_date
    portfolio-tracker | 2020-12-16 15:31:19     prices_df = self.curr_positions(
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/portfolio.py", line 392, in curr_positions
    portfolio-tracker | 2020-12-16 15:31:19     prices_df = data.get_price_data(
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/utils/data.py", line 75, in get_price_data
    portfolio-tracker | 2020-12-16 15:31:19     with ThreadPool(processes=10) as pool:
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    portfolio-tracker | 2020-12-16 15:31:19     Pool.__init__(self, processes, initializer, initargs)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    portfolio-tracker | 2020-12-16 15:31:19     self._change_notifier = self._ctx.SimpleQueue()
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    portfolio-tracker | 2020-12-16 15:31:19     return SimpleQueue(ctx=self.get_context())
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    portfolio-tracker | 2020-12-16 15:31:19     self._rlock = ctx.Lock()
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    portfolio-tracker | 2020-12-16 15:31:19     return Lock(ctx=self.get_context())
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
    portfolio-tracker | 2020-12-16 15:31:19     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
    portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    portfolio-tracker | 2020-12-16 15:31:19     sl = self._semlock = _multiprocessing.SemLock(
    portfolio-tracker | 2020-12-16 15:31:19 OSError: [Errno 38] Function not implemented
    

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
3 answers

I got the same error and suspect that multiprocessing is not support at DO Apps. It is a similar limitation as for AWS Lambdas.

has anyone found a solution to this yet. please its driving me crazy

👋🏼 @gerardsyd @audachny @chidozie

Multiprocessing pools use a syscall that is not supported by App Platform at this time. However this is something that we are working on and will be addressing soon. Unfortunately there is no ETA that I can share.