Celery multiprocessing issues on ubuntu based image

DO’s app platform fails to start up my celery app in prefork pool when using ubunut based image. From python docker repo I used python:3.8-slim. During deployment step I can see the workers throw the following error every time.

Traceback (most recent call last):
  File \"/opt/venv/lib/python3.8/site-packages/celery/worker/\", line 203, in start
  File \"/opt/venv/lib/python3.8/site-packages/celery/\", line 116, in start
  File \"/opt/venv/lib/python3.8/site-packages/celery/\", line 365, in start
    return self.obj.start()
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/\", line 129, in start
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/\", line 109, in on_start
    P = self._pool = Pool(processes=self.limit,
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/\", line 463, in __init__
    super().__init__(processes, *args, **kwargs)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 1046, in __init__
  File \"/opt/venv/lib/python3.8/site-packages/celery/concurrency/\", line 480, in _create_worker_process
    return super()._create_worker_process(i)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 1142, in _create_worker_process
    on_ready_counter = self._ctx.Value('i')
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 181, in Value
    return Value(typecode_or_type, *args, lock=lock,
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 83, in Value
    lock = ctx.RLock()
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 110, in RLock
    return RLock(ctx=self.get_context())
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 207, in __init__
    SemLock.__init__(self, RECURSIVE_MUTEX, 1, 1, ctx=ctx)
  File \"/opt/venv/lib/python3.8/site-packages/billiard/\", line 71, in __init__
    sl = self._semlock = _billiard.SemLock(
OSError: [Errno 38] Function not implemented

For testing purposes I switched to python:3.8-alpine and was able to run the app with no issues, but for my current application apline distro is not suitable. How can I resolve this issue without specifying --pool=gevent for celery ?

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

Try DigitalOcean for free

Click below to sign up and get $200 of credit to try our products over 60 days!

Sign up