Question

Bad gateway error when loading large sklearn model. Django/Ubuntu/Gunicorn/Nginx

Posted January 28, 2019 1.1k views
Nginx Django Applications Ubuntu 18.04

Dear community,

I just developed a machine learning driven python 2.7 application which essentially loads a sklearn model with joblib and predicts a given dataset. On local host the application works perfectly, but when I deploy the django application on digital ocean (Ubuntu 18., gunicorn, nginx) I get a bad gateway error. I found out that the error is being raised when the model is being loaded with joblib. What I noticed as well is that the application works perfectly when a model that has the size of 8000 bites is being loaded but that but that the bad gateway error is being raised when I try to load a model of 60000 bites.

What I have tried is to resize the droplet in order to avoid that the error is being caused by a lack of computing resources, but I still get the error!

Unfortunately, I have no clue how to tackle the problem. So I hope that you guys can help me as maybe anyone here has experienced the same issue before or knows how this problem could be analysed and resolved.

I appreciate any help

Kind regards

Marcel

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

1 answer

Hi Marcel,

Sounds like an nginx timeout error because of the load time of the model. Might have to increase the 504 error time. I am not too sure on the actual issue but first link on google: https://stackoverflow.com/questions/35810749/nginx-504-gateway-timeout-error-for-django

Let us know if that works. Thanks 😃

  • Hey rsivaprasad,

    thanks for your support.

    I added, as described on stack overflow, the following lines to the /etc/ninx/nginx.conf file.

    http {
    proxyconnecttimeout 10;
    proxysendtimeout 15;
    proxyreadtimeout 20;
    }

    unfortunately, this did not solve the issue and the bad gateway error is still being raised.

    In the nginx error.log the following error is being shown: upstream prematurely closed connection while reading response header from upstream.

    The bad gateway error is btw being raised approximately 1-2 seconds after the url is being requested.

    I would appreciate any further hints how to solve this issue as I think that more people will encounter this issue in the future and only very little threats are available on this issue.

    Thanks for your help and kind regards

    Marcel

    • Hmm.. hard to say without seeing the actual code to debug Marcel. Can you go to the console and copy and paste the stack trace here? It might give a hint

Submit an Answer