Join 1M+ other developers and:
- Get help and share knowledge in Q&A
- Subscribe to topics of interest
- Get courses & tools that help you grow as a developer or small business owner
Celery workers dying when running multiple workers or concurrency in Docker
We have docker containers running Django, Celery, Celery Beat and Celery Flower. Recently we have an issue whereby we can only run a single worker with a concurrency of 1, or the worker simply dies when it encounters a task that produces a stacktrace.
For some reason, running a single worker with concurrency of 1 will never die, but as soon as you deviate from this, all workers will die on any stacktrace.
This does not happen on local machine at all, and all docker containers have seemingly enough RAM (we have a 16GB droplet). We do run on a shared CPU, not sure if that is the issue. We were thinking of moving to a new droplet with dedicated CPU and memory optimized, thinking it might be that?