Question
How to configure NGINX to avoid 502 Bad Gateway when communicates with a Docker container?
Hi, my case is that after having set up a new Droplet with an Ubuntu 18.04 server and have followed the DO tutorials to:
- Install Nginx
- Secure Nginx with Let’s Encrypt
- Install Jenkins
- Configure Jenkins with an SSL behind an Nginx reverse proxy
- Set up continuous integration pipeline with Jenkins (this doc says for Ubuntu 16.04)
- Install docker
So the goal is to manage the deployment of a React application to my testdomain.com later, by now, I just want to create the dist folder generated, after the ‘npm build’, within the /var/lib/jenkins/workspace/ , just that. For the purpose, I will run it in a Jenkins pipeline with docker.
So by now, I’m able to access my jenkins.testdomain.com alright, trigger the pipeline to start the process after pushing to my repo, and start to run the stages; but it’s here when start to fail nginx, when the pipeline reaches the Deliver phase (let’s read ‘npm build’ phase).
It’s at this point reading the Jenkins console output I see when it gets stuck in that phase to eventually show a 502 Bad Gateway.
After this, I will need to make a systemctl restart Jenkins to have access again. After restarting the pipeline resume the work and seems to get the job done :/
In the /var/log/nginx/error.log for nginx I can read:
*1 connect() failed (111: Connection refused) while connecting to upstream, client: 85.146.85.194, server: jenkins.testdomain.com, request: “GET /job/Basic%20NodeJS-React%20app/8/console HTTP/1.1”, upstream: “https://127.0.0.1:8080/job/Basic NodeJS-React app/8/console”, host: “jenkins.testdomain.com”, referrer: “https://jenkins.testdomain.com/job/Basic NodeJS-React app/8/”
*1 SSL_do_handshake() failed (SSL: error:1408F10B:SSL routines:ssl3_get_record:wrong version number) while SSL handshaking to upstream, client: 85.146.85.194, server: jenkins.testdomain.com, request: “GET /favicon.ico HTTP/1.1”, upstream: “https://127.0.0.1:8080/favicon.ico”, host: “jenkins.testdomain.com”, referrer: “https://jenkins.testdomain.com/job/Basic NodeJS-React app/8/console” …
In the Jenkinsfile of my node-js-react app, the agent looks like this (pass tests from nginx -t)
pipeline {
agent {
docker {
image 'node:6-alpine'
args '-p 3000:80'
}
}
environment {
CI = 'true'
}
stages {
// Build, Test, and Deliver stages
}
}
And my jenkins.testdomain.com configuration (/etc/nginx/sites-available/jenkins.testdomain.com) is like this:
server {
listen 80;
root /var/www/jenkins.testdomain.com/html;
server_name jenkins.testdomain.com www.jenkins.testdomain.com;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
# Fix the "It appears that your reverse proxy set up is broken" error.
proxy_pass http://localhost:8080;
# High timeout for testing
proxy_connect_timeout 1200s;
proxy_send_timeout 1200s;
proxy_read_timeout 1200s;
proxy_redirect http://localhost:8080 https://jenkins.testdomain.com;
# Required for new HTTP-based CLI
proxy_http_version 1.1;
proxy_request_buffering off;
# Required for HTTP-based CLI to work over SSL
proxy_buffering off;
}
# Certbot auto-generated lines...
}
Please, any help on this it would really helpful, after 2-3 days I’ve tried several things to avoid that error, but it gets back at some point.
Thanks in advance!
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
OK just add an update that some days after my latest post, I realized that the main and only reason the server was going down was a lack of resources in the droplet. So I was using a droplet with 1GB of RAM, 25GB HD, etc… (the most basic one), so I chose to update it to use at least 2GB of RAM and indeed, that made it work as I was expecting. Everything until now works fine and that issue didn’t happen again.
Hope it helps if someone experiences the same issue.
I am having a similar issue. We have many jenkins masters for different BUs but all with same verion of jenkins and configs. Only for some, Jenkins just start throwing 502 error when accessing URL. This happens once in 2 weeks or so and also to few of the jenkins installations. Even, nothing shows up in the jenkins.log untill a restart of jenkins service. And then works fine till when it next gets stuck. I have also noted that this mostly happens when there is no jobs running for some prolonged time.
To anybody else ending here, make sure to doublecheck wether the variables are with - or _
Nginx for kubernetes uses - in configmap etc, so might be
Hi,
After checking it several times, it looks like that the react-nodejs project I’m trying, it’s intermittently failing. Some times complete the whole task successfully, but in other cases, it just seems to stop the Deploy process and after some seconds (let’s say around one minute as most) it ends up showing the 502 Bad Gateway error.
Other nodejs-react project I’m trying (with my own custom webpack, which it creates the build locally without issues), it always gets the error in the ‘npm build’ phase or sometimes even earlier (‘npm install’ phase). But again, when I restart Jenkins it resumes and gets the job done!
I’ve already tried playing around with the memory from Jenkins if that would be the case, but with no difference perceived with these changes:
JAVA_ARGS="-Xmx256m"orJAVA_ARGS="-Xmx2048m -XX:MaxPermSize=512m"kernel: [ 1716.509477] docker0: port 1(vethcf5108c) entered blocking state kernel: [ 1716.509479] docker0: port 1(vethcf5108c) entered disabled state kernel: [ 1716.509606] device vethcf5108c entered promiscuous mode systemd-timesyncd[535]: Network configuration changed, trying to establish connection. networkd-dispatcher[779]: WARNING:Unknown index 13 seen, reloading interface list kernel: [ 1716.522790] IPv6: ADDRCONF(NETDEV_UP): vethcf5108c: link is not ready systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). systemd-udevd[5472]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. systemd-udevd[5472]: Could not generate persistent MAC address for vethc118367: No such file or directory systemd-udevd[5474]: link_config: autonegotiation is unset or enabled, the speed and duplex are not writable. systemd-udevd[5474]: Could not generate persistent MAC address for vethcf5108c: No such file or directory systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). containerd[811]: time="2019-10-14T10:48:04.960106015+02:00" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/b1ae788b18bd4dfd5fd7870cc194dd0bbee8591ee5402846cc4030945d8794de/shim.sock" debug=false pid=5485 systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). kernel: [ 1717.167230] eth0: renamed from vethc118367 kernel: [ 1717.168525] IPv6: ADDRCONF(NETDEV_CHANGE): vethcf5108c: link becomes ready kernel: [ 1717.168564] docker0: port 1(vethcf5108c) entered blocking state kernel: [ 1717.168566] docker0: port 1(vethcf5108c) entered forwarding state systemd-networkd[592]: vethcf5108c: Gained carrier systemd-networkd[592]: docker0: Gained carrier systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). systemd-networkd[592]: vethcf5108c: Gained IPv6LL systemd-timesyncd[535]: Network configuration changed, trying to establish connection. systemd-timesyncd[535]: Synchronized to time server 91.189.94.4:123 (ntp.ubuntu.com). kernel: [ 1719.454959] [UFW BLOCK] IN=eth0 OUT= MAC=MAC_ADDRESS SRC=89.248.162.168 DST=XX.XX.XX.XX LEN=40 TOS=0x00 PREC=0x00 TTL=252 ID=42701 PROTO=TCP SPT=40602 DPT=3580 WINDOW=1024 RES=0x00 SYN URGP=0Thanks in advance.