Question

Connection refused while connecting to upstream, client & 502 gateway errors

I posted this question originally on the CompreFace GitHub page as I initially thought the issue was with their product, however, I have been able to install the CompreFace applications in a docker container on my local machine and on a competitor’s (Vultr) hosted docker instance.

DigitalOcean appears to be preventing the (4 applications and 1 database that make up the) CompreFace identification system from talking to itself.

Below is the contents of the error I logged with CompreFace. The problem though is with DigitalOcean and I cannot figure out what to change to be able to get the various apps in this container to talk to each other.

Below are steps to reproduce and the error logs on the docker instance. When I attempt to access thepage using the IP address the API calls all return 502 bad gateway error on the client.

Photo of running images

The error below appears several times “connect() failed (111: Connection refused) while connecting to upstream, client…”

compreface-ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration compreface-ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh compreface-ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh compreface-ui | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh compreface-ui | /docker-entrypoint.sh: Configuration complete; ready for start up compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:39 +0000] "GET /sign-up HTTP/1.1" 200 1077 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:40 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up" compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:40 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/sign-up" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:43 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/login" compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:43 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:45 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up"

And when I click on Sign Up these are the errors produced.

compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:45 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/sign-up" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:52:11 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "POST /admin/user/register HTTP/1.1", upstream: "http://172.20.0.5:8080/user/register", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up"

To Reproduce Steps to reproduce the behavior:

1.) Create a new droplet on DigitalOcean with Docker 19.03.12 on Ubuntu 20.4 (there is a pre-existing install). This also installed Docker Compose by default.

2.) Download CompreFace_1.1.0.zip archive or run:

wget -q -O tmp.zip 'https://github.com/exadel-inc/CompreFace/releases/download/v1.1.0/CompreFace_1.1.0.zip' && unzip tmp.zip && rm tmp.zip

3.) To start CompreFace run:

docker-compose up -d

4.) Open in your browser: http://:8000/login

5.) Click on Create an account

6.) Enter your details and click Sign Up

To reiterate. The above works flawlessly on a Vultr instance or a local docker instance. There is something strange about the DigitalOcean setup. I found one forum post that described a similar problem which they put down to a configuration issue at Digital Ocean that prevents containers in the same instance from talking to each other. How do I get around this?

I have switched ufw firewall off completely and that did not solve the problem.

Show comments

Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Hi there,

I’ve just tested this out on a fresh new Droplet and it seems to be working as expected:

Note, that I was able to replicate the 502 error as well, but it was only when visiting http://:8000/login right after the docker-compose up -d completed. According to the CompreFace docs, you need to wait a couple of minutes until all services are up and running, because if you stop it during this time, as it may corrupt database data during data migration.

Another thing that comes to my mind is, you server might be low on memory, you could add a swap file to have some extra buffer:

https://www.digitalocean.com/community/tutorials/how-to-add-swap-space-on-ubuntu-22-04

Best,

Bobby