Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
How to migrate from Python buildpack to Dockerfile without creating a new service? Question Question
Problem running python/Jinja as a build step for a static site. Question Question

Question

App Platform multiprocessing (Python)

Posted December 16, 2020 1.4k views
PythonDigitalOcean App Platform

I’ve managed to deploy my app onto the App platform but when it runs, it now gives me an errno 38 Function not implemented?
Looking through the tracestack, this looks related to using multiprocessing pools in my app. Is this not supported by the App platform?

Add a comment
gerardsyd
By:
gerardsyd
edited by MattIPv4

Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
How to migrate from Python buildpack to Dockerfile without creating a new service? Question Question
Problem running python/Jinja as a build step for a static site. Question Question
Add a comment
1 comment
  • gerardsyd December 16, 2020
    Reply Report

    Full Traceback as below

    portfolio-tracker | 2020-12-16 13:04:30 Traceback (most recent call last):
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
portfolio-tracker | 2020-12-16 13:04:30     self.handle_request(listener, req, client, addr)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 190, in handle_request
portfolio-tracker | 2020-12-16 13:04:30     util.reraise(*sys.exc_info())
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/util.py", line 625, in reraise
portfolio-tracker | 2020-12-16 13:04:30     raise value
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
portfolio-tracker | 2020-12-16 13:04:30     respiter = self.wsgi(environ, resp.start_response)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
portfolio-tracker | 2020-12-16 13:04:30     return self.wsgi_app(environ, start_response)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
portfolio-tracker | 2020-12-16 13:04:30     response = self.handle_exception(e)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
portfolio-tracker | 2020-12-16 13:04:30     reraise(exc_type, exc_value, tb)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
portfolio-tracker | 2020-12-16 13:04:30     raise value
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
portfolio-tracker | 2020-12-16 13:04:30     response = self.full_dispatch_request()
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
portfolio-tracker | 2020-12-16 13:04:30     rv = self.handle_user_exception(e)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
portfolio-tracker | 2020-12-16 13:04:30     reraise(exc_type, exc_value, tb)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
portfolio-tracker | 2020-12-16 13:04:30     raise value
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
portfolio-tracker | 2020-12-16 13:04:30     rv = self.dispatch_request()
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
portfolio-tracker | 2020-12-16 13:04:30     return self.view_functions[rule.endpoint](**req.view_args)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
portfolio-tracker | 2020-12-16 13:04:30     return func(*args, **kwargs)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/routes.py", line 32, in index
portfolio-tracker | 2020-12-16 13:04:30     return update_pf()
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
portfolio-tracker | 2020-12-16 13:04:30     return func(*args, **kwargs)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/routes.py", line 95, in update_pf
portfolio-tracker | 2020-12-16 13:04:30     df = pf.info_date(as_at_date, hide_zero_pos=hide_zero, no_update=no_update)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/portfolio.py", line 114, in info_date
portfolio-tracker | 2020-12-16 13:04:30     prices_df = self.curr_positions(
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/app/portfolio.py", line 392, in curr_positions
portfolio-tracker | 2020-12-16 13:04:30     prices_df = data.get_price_data(
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/utils/data.py", line 75, in get_price_data
portfolio-tracker | 2020-12-16 13:04:30     with ThreadPool(processes=10) as pool:
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
portfolio-tracker | 2020-12-16 13:04:30     Pool.__init__(self, processes, initializer, initargs)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
portfolio-tracker | 2020-12-16 13:04:30     self._change_notifier = self._ctx.SimpleQueue()
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
portfolio-tracker | 2020-12-16 13:04:30     return SimpleQueue(ctx=self.get_context())
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
portfolio-tracker | 2020-12-16 13:04:30     self._rlock = ctx.Lock()
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 68, in Lock
portfolio-tracker | 2020-12-16 13:04:30     return Lock(ctx=self.get_context())
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
portfolio-tracker | 2020-12-16 13:04:30     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
portfolio-tracker | 2020-12-16 13:04:30   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
portfolio-tracker | 2020-12-16 13:04:30     sl = self._semlock = _multiprocessing.SemLock(
portfolio-tracker | 2020-12-16 13:04:30 OSError: [Errno 38] Function not implemented
portfolio-tracker | 2020-12-16 15:31:19 [2020-12-16 04:31:19 +0000] [12] [ERROR] Socket error processing request.
portfolio-tracker | 2020-12-16 15:31:19 Traceback (most recent call last):
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 134, in handle
portfolio-tracker | 2020-12-16 15:31:19     self.handle_request(listener, req, client, addr)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 190, in handle_request
portfolio-tracker | 2020-12-16 15:31:19     util.reraise(*sys.exc_info())
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/util.py", line 625, in reraise
portfolio-tracker | 2020-12-16 15:31:19     raise value
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/gunicorn/workers/sync.py", line 175, in handle_request
portfolio-tracker | 2020-12-16 15:31:19     respiter = self.wsgi(environ, resp.start_response)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
portfolio-tracker | 2020-12-16 15:31:19     return self.wsgi_app(environ, start_response)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
portfolio-tracker | 2020-12-16 15:31:19     response = self.handle_exception(e)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
portfolio-tracker | 2020-12-16 15:31:19     reraise(exc_type, exc_value, tb)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
portfolio-tracker | 2020-12-16 15:31:19     raise value
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
portfolio-tracker | 2020-12-16 15:31:19     response = self.full_dispatch_request()
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
portfolio-tracker | 2020-12-16 15:31:19     rv = self.handle_user_exception(e)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
portfolio-tracker | 2020-12-16 15:31:19     reraise(exc_type, exc_value, tb)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
portfolio-tracker | 2020-12-16 15:31:19     raise value
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
portfolio-tracker | 2020-12-16 15:31:19     rv = self.dispatch_request()
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
portfolio-tracker | 2020-12-16 15:31:19     return self.view_functions[rule.endpoint](**req.view_args)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
portfolio-tracker | 2020-12-16 15:31:19     return func(*args, **kwargs)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/routes.py", line 32, in index
portfolio-tracker | 2020-12-16 15:31:19     return update_pf()
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/site-packages/flask_login/utils.py", line 272, in decorated_view
portfolio-tracker | 2020-12-16 15:31:19     return func(*args, **kwargs)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/routes.py", line 95, in update_pf
portfolio-tracker | 2020-12-16 15:31:19     df = pf.info_date(as_at_date, hide_zero_pos=hide_zero, no_update=no_update)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/portfolio.py", line 114, in info_date
portfolio-tracker | 2020-12-16 15:31:19     prices_df = self.curr_positions(
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/app/portfolio.py", line 392, in curr_positions
portfolio-tracker | 2020-12-16 15:31:19     prices_df = data.get_price_data(
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/utils/data.py", line 75, in get_price_data
portfolio-tracker | 2020-12-16 15:31:19     with ThreadPool(processes=10) as pool:
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
portfolio-tracker | 2020-12-16 15:31:19     Pool.__init__(self, processes, initializer, initargs)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
portfolio-tracker | 2020-12-16 15:31:19     self._change_notifier = self._ctx.SimpleQueue()
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
portfolio-tracker | 2020-12-16 15:31:19     return SimpleQueue(ctx=self.get_context())
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
portfolio-tracker | 2020-12-16 15:31:19     self._rlock = ctx.Lock()
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/context.py", line 68, in Lock
portfolio-tracker | 2020-12-16 15:31:19     return Lock(ctx=self.get_context())
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 162, in __init__
portfolio-tracker | 2020-12-16 15:31:19     SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
portfolio-tracker | 2020-12-16 15:31:19   File "/workspace/.heroku/python/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
portfolio-tracker | 2020-12-16 15:31:19     sl = self._semlock = _multiprocessing.SemLock(
portfolio-tracker | 2020-12-16 15:31:19 OSError: [Errno 38] Function not implemented

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
4 answers
audachny February 7, 2021

I got the same error and suspect that multiprocessing is not support at DO Apps. It is a similar limitation as for AWS Lambdas.

Reply Report
chidozie April 29, 2021

has anyone found a solution to this yet. please its driving me crazy

Reply Report
kamaln7 April 30, 2021

👋🏼 @gerardsyd @audachny @chidozie

Multiprocessing pools use a syscall that is not supported by App Platform at this time. However this is something that we are working on and will be addressing soon. Unfortunately there is no ETA that I can share.

Reply Report
dennisXiloj June 2, 2021

I managed to solve it by ditching the multiprocessing in favor of gevent.

But in my case i was trying to install a Celery worker with prefork pools (which are multitreaded).

I switched from running:

celery -A app worker

to run: 

celery -A app worker -P gevent --concurrency=100

Not sure if you can also switch to gevent (or greenlet) and if doing so will benefit you, as gevent isn’t meant for CPU intensive tasks, but is fine in my case because it need it mainly for making a lot of http requests

Reply Report

Related

Looking for something else?

Ask a new question Search for more help