Bad gateway error when loading large sklearn model. Django/Ubuntu/Gunicorn/Nginx

January 28, 2019 614 views
Django Nginx Applications Ubuntu 18.04

Dear community,

I just developed a machine learning driven python 2.7 application which essentially loads a sklearn model with joblib and predicts a given dataset. On local host the application works perfectly, but when I deploy the django application on digital ocean (Ubuntu 18., gunicorn, nginx) I get a bad gateway error. I found out that the error is being raised when the model is being loaded with joblib. What I noticed as well is that the application works perfectly when a model that has the size of 8000 bites is being loaded but that but that the bad gateway error is being raised when I try to load a model of 60000 bites.

What I have tried is to resize the droplet in order to avoid that the error is being caused by a lack of computing resources, but I still get the error!

Unfortunately, I have no clue how to tackle the problem. So I hope that you guys can help me as maybe anyone here has experienced the same issue before or knows how this problem could be analysed and resolved.

I appreciate any help

Kind regards

Marcel

1 Answer

Hi Marcel,

Sounds like an nginx timeout error because of the load time of the model. Might have to increase the 504 error time. I am not too sure on the actual issue but first link on google: https://stackoverflow.com/questions/35810749/nginx-504-gateway-timeout-error-for-django

Let us know if that works. Thanks 😃

  • Hey rsivaprasad,

    thanks for your support.

    I added, as described on stack overflow, the following lines to the /etc/ninx/nginx.conf file.

    http {
    proxyconnecttimeout 10;
    proxysendtimeout 15;
    proxyreadtimeout 20;
    }

    unfortunately, this did not solve the issue and the bad gateway error is still being raised.

    In the nginx error.log the following error is being shown: upstream prematurely closed connection while reading response header from upstream.

    The bad gateway error is btw being raised approximately 1-2 seconds after the url is being requested.

    I would appreciate any further hints how to solve this issue as I think that more people will encounter this issue in the future and only very little threats are available on this issue.

    Thanks for your help and kind regards

    Marcel

    • Hmm.. hard to say without seeing the actual code to debug Marcel. Can you go to the console and copy and paste the stack trace here? It might give a hint

Have another answer? Share your knowledge.