Question

Celery multiprocessing issues on ubuntu based image

DO’s app platform fails to start up my celery app in prefork pool when using ubunut based image. From python docker repo I used python:3.8-slim. During deployment step I can see the workers throw the following error every time.

Traceback (most recent call last):
  File \"/opt/venv/lib/python3.8/site-packages/celery/worker/worker.py\", line 203, in start
    self.blueprint.start(self)
  File \"/opt/venv/lib/python3.8/site-packages/celery/bootsteps.py\", line 116, in start
    step.start(parent)
  File \"/opt/venv/lib/python3.8/site-packages/celery/bootsteps.py\", line 365, in start
    return self.obj.start()
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/base.py\", line 129, in start
    self.on_start()
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/prefork.py\", line 109, in on_start
    P = self._pool = Pool(processes=self.limit,
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py\", line 463, in __init__
    super().__init__(processes, *args, **kwargs)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/pool.py\", line 1046, in __init__
    self._create_worker_process(i)
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/asynpool.py\", line 480, in _create_worker_process
    return super()._create_worker_process(i)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/pool.py\", line 1142, in _create_worker_process
    on_ready_counter = self._ctx.Value('i')
  File \"/opt/venv/lib/python3.8/site-packages/billiard/context.py\", line 181, in Value
    return Value(typecode_or_type, *args, lock=lock,
  File \"/opt/venv/lib/python3.8/site-packages/billiard/sharedctypes.py\", line 83, in Value
    lock = ctx.RLock()
  File \"/opt/venv/lib/python3.8/site-packages/billiard/context.py\", line 110, in RLock
    return RLock(ctx=self.get_context())
  File \"/opt/venv/lib/python3.8/site-packages/billiard/synchronize.py\", line 207, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/synchronize.py\", line 71, in __init__
    sl = self._semlock = _billiard.SemLock(
OSError: [Errno 38] Function not implemented

For testing purposes I switched to python:3.8-alpine and was able to run the app with no issues, but for my current application apline distro is not suitable. How can I resolve this issue without specifying --pool=gevent for celery ?


Submit an answer


This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up