By Arthur
DO’s app platform fails to start up my celery app in prefork pool when using ubunut based image. From python docker repo I used python:3.8-slim. During deployment step I can see the workers throw the following error every time.
Traceback (most recent call last):
File \"/opt/venv/lib/python3.8/site-packages/celery/worker/worker.py\", line 203, in start
self.blueprint.start(self)
File \"/opt/venv/lib/python3.8/site-packages/celery/bootsteps.py\", line 116, in start
step.start(parent)
File \"/opt/venv/lib/python3.8/site-packages/celery/bootsteps.py\", line 365, in start
return self.obj.start()
File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/base.py\", line 129, in start
self.on_start()
File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/prefork.py\", line 109, in on_start
P = self._pool = Pool(processes=self.limit,
File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py\", line 463, in __init__
super().__init__(processes, *args, **kwargs)
File \"/opt/venv/lib/python3.8/site-packages/billiard/pool.py\", line 1046, in __init__
self._create_worker_process(i)
File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py\", line 480, in _create_worker_process
return super()._create_worker_process(i)
File \"/opt/venv/lib/python3.8/site-packages/billiard/pool.py\", line 1142, in _create_worker_process
on_ready_counter = self._ctx.Value('i')
File \"/opt/venv/lib/python3.8/site-packages/billiard/context.py\", line 181, in Value
return Value(typecode_or_type, *args, lock=lock,
File \"/opt/venv/lib/python3.8/site-packages/billiard/sharedctypes.py\", line 83, in Value
lock = ctx.RLock()
File \"/opt/venv/lib/python3.8/site-packages/billiard/context.py\", line 110, in RLock
return RLock(ctx=self.get_context())
File \"/opt/venv/lib/python3.8/site-packages/billiard/synchronize.py\", line 207, in __init__
SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
File \"/opt/venv/lib/python3.8/site-packages/billiard/synchronize.py\", line 71, in __init__
sl = self._semlock = _billiard.SemLock(
OSError: [Errno 38] Function not implemented
For testing purposes I switched to python:3.8-alpine and was able to run the app with no issues, but for my current application apline distro is not suitable. How can I resolve this issue without specifying --pool=gevent for celery ?
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hi there,
I have seen a similar question about this here:
https://www.digitalocean.com/community/questions/is-there-an-issue-with-celery-on-app-platform
The answer for the DigitalOcean engineer was:
Thanks for that info. Looking into this further it looks like Celery uses multiprocessing pools that use a syscall that is not supported by App Platform at this time. However, this is something that we are working on and will be addressing soon. Unfortunately, there is no ETA that I can share.
The solution that most users used was to bypass this by setting Celery’s pool handler to gevent as you mentioned.
I’ve forwarded this feedback internally to the team again.
- Bobby
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.