Nginx docker container exits with error “fopen:No such file or directory:fopen('/etc/nginx/ssl/live/test.example.dev/fullchain.pem'” for SSL

Okay so I’m learning Docker and I am trying to deploy a test app with a subdomain (who’s domain was bought from another provider) which is pointing to my server. The server already has non-dockerized Nginx setup which serves couple of other non-dockerized apps perfectly. And that part means Nginx is already using port 80 and 443. And the subdomain somehow shows Nginx default page when visited. This is my server situation. Now let me explain what happens with the dockerized app.

The app is using 4 images to create 4 containers: Nodejs, Mongodb, Nginx and Certbot. Before adding Certbot, I could perfectly access the app with <server-ip>:<port>. But now I need to attach that subdomain (test.example.dev) to my app with Let’s Encrypt SSL certificates.

So after the build is done with Docker Compose, I see that Nginx and Certbot is exited with errors.

This is my nginx/default.conf file:

server {
    listen 80;
    listen [::]:80;

    server_name test.example.dev;
    server_tokens off;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://test.example.dev$request_uri;
    }
}

server {
    listen 443 default_server ssl http2;
    listen [::]:443 ssl http2;

    server_name test.example.dev;

    ssl_certificate /etc/nginx/ssl/live/test.example.dev/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/live/test.example.dev/privkey.pem;
    
    location /api {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://practice-app:3050;
        proxy_redirect off;

    }
}

Here’s my shared docker-compose file for both dev and prod environment:

version: '3'
services:

  practice-app:
    build: .
    depends_on:
      - mongo
      
  nginx: 
    image: nginx:stable-alpine
    ports: 
      - "4088:80"
    volumes: 
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - practice-app

  mongo:
    image: mongo:4.4.6
    environment:
      - MONGO_INITDB_ROOT_USERNAME=test
      - MONGO_INITDB_ROOT_PASSWORD=test
    volumes:
      - mongo-db:/data/db
      
volumes:
  mongo-db:

And here’s my docker-compose.prod.yml file:

version: "3"
services: 

  practice-app:
    build: 
      context: .
      args: 
        NODE_ENV: production
    environment:
      - NODE_ENV=production
    command: node index.js

  nginx: 
    ports: 
      - "4088:80"
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
      - ./certbot/www:/var/www/certbot/:ro
      - ./certbot/conf/:/etc/nginx/ssl/:ro

  certbot:
    image: certbot/certbot:latest
    volumes:
      - ./certbot/www/:/var/www/certbot/:rw
      - ./certbot/conf/:/etc/letsencrypt/:rw
    depends_on:
      - nginx
      
  mongo: 
    environment:
      - MONGO_INITDB_ROOT_USERNAME=test
      - MONGO_INITDB_ROOT_PASSWORD=test 

Nginx logs says:

/docker-entrypoint.sh: Configuration complete; ready for start up
2022/01/31 13:42:28 [emerg] 1#1: cannot load certificate "/etc/nginx/ssl/live/test.example.dev/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/nginx/ssl/live/test.example.dev/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/nginx/ssl/live/test.example.dev/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/nginx/ssl/live/test.example.dev/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

And Certbot logs says:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Certbot doesn't know how to automatically configure the web server on this system. However, it can still get a certificate for you. Please run "certbot certonly" to do so. You'll need to manually configure your web server to use the resulting certificate.

But after adding the following code:

command: certonly --webroot -w /var/www/certbot --force-renewal --email example@gmail.com -d test.example.dev --agree-tos

under certbot service, the log changed to this:

[17:00] [server1.com test] # docker logs test_certbot_1

Saving debug log to /var/log/letsencrypt/letsencrypt.log Skipped user interaction because Certbot doesn't appear to be running in a terminal. You should probably include --non-interactive or --force-interactive on the command line. 

Account registered. 
Requesting a certificate for test.example.dev

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:

Domain: test.example.dev 
Type:   unauthorized 
Detail: Invalid response from http://test.example.dev/.well-known/acme-challenge/J7_MZAg5yYLi3P_VVEalOwlU5cH3NAmevlhTKCvt6Zo [xx.xx0.xxx.7x]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed. Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

What am I doing wrong here? Please give me a beginner friendly solution as I am new to DevOps.