Connection refused while connecting to upstream, client & 502 gateway errors

I posted this question originally on the CompreFace GitHub page as I initially thought the issue was with their product, however, I have been able to install the CompreFace applications in a docker container on my local machine and on a competitor’s (Vultr) hosted docker instance.

DigitalOcean appears to be preventing the (4 applications and 1 database that make up the) CompreFace identification system from talking to itself.

Below is the contents of the error I logged with CompreFace. The problem though is with DigitalOcean and I cannot figure out what to change to be able to get the various apps in this container to talk to each other.

Below are steps to reproduce and the error logs on the docker instance. When I attempt to access thepage using the IP address the API calls all return 502 bad gateway error on the client.

The error below appears several times “connect() failed (111: Connection refused) while connecting to upstream, client…”

compreface-ui | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration compreface-ui | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/ compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh compreface-ui | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh compreface-ui | 20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/nginx.conf.template to /etc/nginx/conf.d/nginx.conf compreface-ui | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh compreface-ui | /docker-entrypoint.sh: Configuration complete; ready for start up compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:39 +0000] "GET /sign-up HTTP/1.1" 200 1077 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:40 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up" compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:40 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/sign-up" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:43 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/login" compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:43 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/login" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:51:45 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "GET /api/v1/consistence/status HTTP/1.1", upstream: "http://172.20.0.4:8080/api/v1/consistence/status", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up"

And when I click on Sign Up these are the errors produced.

compreface-ui | 151.210.164.7 - - [19/Oct/2022:05:51:45 +0000] "GET /api/v1/consistence/status HTTP/1.1" 502 150 "http://<my-ip-address>:8000/sign-up" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Safari/605.1.15" compreface-ui | 2022/10/19 05:52:11 [error] 27#27: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 151.210.164.7, server: ui, request: "POST /admin/user/register HTTP/1.1", upstream: "http://172.20.0.5:8080/user/register", host: "<my-ip-address>:8000", referrer: "http://<my-ip-address>:8000/sign-up"

To Reproduce Steps to reproduce the behavior:

1.) Create a new droplet on DigitalOcean with Docker 19.03.12 on Ubuntu 20.4 (there is a pre-existing install). This also installed Docker Compose by default.

2.) Download CompreFace_1.1.0.zip archive or run:

wget -q -O tmp.zip 'https://github.com/exadel-inc/CompreFace/releases/download/v1.1.0/CompreFace_1.1.0.zip' && unzip tmp.zip && rm tmp.zip

3.) To start CompreFace run:

docker-compose up -d

4.) Open in your browser: http://:8000/login

5.) Click on Create an account

6.) Enter your details and click Sign Up

To reiterate. The above works flawlessly on a Vultr instance or a local docker instance. There is something strange about the DigitalOcean setup. I found one forum post that described a similar problem which they put down to a configuration issue at Digital Ocean that prevents containers in the same instance from talking to each other. How do I get around this?

I have switched ufw firewall off completely and that did not solve the problem.