Why is my droplet not installing nginx through terraform and cloud init

Question

I am provisioning the infrastructure using terraform to get a droplet and a pg db cluster up and running. The droplet frontend is nginx and it’s installed through the user-data providing a cloud-config file. However, it fails to install nginx and seems to not be connected to the internet at all. I don’t really understand as I tried the same cloud config file alone by using the cloud digitalocean dashboard and copy pasting my cloud config file into the user data form item when creating a droplet and it worked. Am I missing something?

My cloud config file

#cloud-config

package_update: true
package_upgrade: false
package_reboot_if_required: false

packages:
  - nginx

write_files:
  - content: |
      user www-data;
      worker_processes auto;
      pid /run/nginx.pid;
      include /etc/nginx/modules-enabled/*.conf;

      events {
        worker_connections 768;
      }

      http {
        server {
          listen    80  proxy_protocol;
          server_name   assessment-test.gaitup.com cloud.digitalocean.com;
          location / {
            proxy_pass       http://localhost:${PORT};
            proxy_set_header X-Real-IP       $proxy_protocol_addr;
            proxy_set_header X-Forwarded-For $proxy_protocol_addr;
            proxy_set_header Host $host;
          }
        }

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        log_format main '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$http_x_forwarded_for"';

        access_log /var/log/nginx/access.log main;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
      }
    path: /etc/nginx/nginx.conf
  - content: |
      PORT=${PORT}
      DATABASE_URL=${DATABASE_URL}
    path: /root/.env

runcmd:
- ufw allow 'Nginx HTTP'
- ufw allow 'Nginx HTTPS'
- systemctl restart nginx

my terraform file for the drople:

resource "digitalocean_droplet" "www-1" {
  image              = "docker-20-04"
  name               = format("%s.com.gaitup.%s.www-1", var.env, var.app_name)
  region             = var.region
  size               = "s-1vcpu-1gb"
  # private_networking = true
  vpc_uuid           = data.digitalocean_vpc.www-vpc.id
  backups            = true
  monitoring         = true
  ssh_keys = [
    data.digitalocean_ssh_key.terraform_do_rsa.id
  ]
  connection {
    host        = self.ipv4_address
    user        = "root"
    type        = "ssh"
    private_key = file(pathexpand("~/${var.pvt_key}"))
    timeout     = "2m"
  }
  user_data = templatefile("${path.root}/www-cloud-init.yaml", { 
    APP_VERSION = var.app_version, 
    APP_NAME = var.app_name,
    DO_API_PERSONAL_TOKEN = var.do_token,
    PORT                  = var.port,
    DATABASE_URL          = format("%s:%d/%s", digitalocean_database_cluster.www-db.private_host, digitalocean_database_cluster.www-db.port, digitalocean_database_db.gaitup_db.name)
  })
}

resource "digitalocean_database_firewall" "www-fw" {
  cluster_id = digitalocean_database_cluster.www-db.id

  rule {
    type  = "droplet"
    value = digitalocean_droplet.www-1.id
  }
}

resource "digitalocean_database_db" "gaitup_db" {
  cluster_id = digitalocean_database_cluster.www-db.id
  name       = format("gaitup_%s", var.env)
}

resource "digitalocean_database_cluster" "www-db" {
  name                 = format("%s-com-gaitup-%s-db", var.env, replace(var.app_name, "/[_\\.]/g", "-"))
  engine               = "pg"
  version              = "12"
  size                 = "db-s-1vcpu-1gb"
  region               = var.region
  node_count           = 1
  private_network_uuid = data.digitalocean_vpc.www-vpc.id
}

resource "digitalocean_firewall" "apos-algo-backend" {
  name = "only-apos-algo-backend"

  droplet_ids = [digitalocean_droplet.www-1.id]

  # inbound_rule {
  #   protocol         = "tcp"
  #   port_range       = "80"
  #   source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
  # }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "80"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "22"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "tcp"
    port_range            = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
}

Now what I have been able to see looking at the log files of cloud init is the following:

Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:config' at Tue, 02 Feb 2021 17:00:07 +0000. Up 18.83 seconds.
Err:1 https://download.docker.com/linux/ubuntu focal InRelease
  Temporary failure resolving 'download.docker.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease  Temporary failure resolving 'download.docker.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package nginx
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:final' at Tue, 02 Feb 2021 17:00:28 +0000. Up 40.53 seconds.
2021-02-02 17:01:20,233 - util.py[WARNING]: Failed to install packages: ['nginx']
2021-02-02 17:01:20,392 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2021-02-02 17:01:20,394 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
curl: (6) Could not resolve host: insights.nyc3.cdn.digitaloceanspaces.com
Initializing machine ID from D-Bus machine ID.
ERROR: Could not find a profile matching 'Nginx HTTP'
ERROR: Could not find a profile matching 'Nginx HTTPS'
Failed to restart nginx.service: Unit nginx.service not found.
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Unable to find image 'registry.digitalocean.com/gaitup/gaitup-live-customer-licensing:0.0.1' locally
docker: Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution.
See 'docker run --help'.
2021-02-02 17:03:51,407 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2021-02-02 17:03:51,408 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 finished at Tue, 02 Feb 2021 17:03:51 +0000. Datasource DataSourceDigitalOcean.  Up 243.16 seconds

It does not seem to have any internet connection. I was able to confirm that by doing in the droplet

$ ping -c 1 google.com # -> it fails with "ping: google.com: Temporary failure in name resolution"

However pinging an IP directly:

$ ping -c 1 4.2.2.1

returns

PING google.com (172.217.168.46): 56 data bytes
64 bytes from 172.217.168.46: icmp_seq=0 ttl=57 time=12.742 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.742/12.742/12.742/0.000 ms

So some issue maybe with DNS maybe?
Trying to nslookup google indeed fails:

$ nslookup google.com
Server:     127.0.0.53
Address:    127.0.0.53#53

** server can't find google.com: SERVFAIL

So I thought to look at the firewall with ufw

$ ufw status
22/tcp                     LIMIT       Anywhere
2375/tcp                   ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
22/tcp (v6)                LIMIT       Anywhere (v6)
2375/tcp (v6)              ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)

Looks like traffic on port 80 and 443 is not allowed. So next move was to allow it:

$ ufw allow HTTP
$ ufw allow HTTPS
$ ufw status
22/tcp                     LIMIT       Anywhere
2375/tcp                   ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
22/tcp (v6)                LIMIT       Anywhere (v6)
2375/tcp (v6)              ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

now it they are allowed. Next I re try a pinging:

$ ping -c 1 google.com
ping: google.com: Temporary failure in name resolution

Last, since the provisioning using cloud-config did not work. I thought at this point let’s give it a try manually. So I ssh into the droplet and start with:

$ sudo apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 https://download.docker.com/linux/ubuntu focal InRelease
  Temporary failure resolving 'download.docker.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease  Temporary failure resolving 'download.docker.com'

Nope.
Nor does:

$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done

Now I am stuck. I also should mention, on another server, using the same approach, with the same cloud-config file, the droplet was able to install nginx in docker 18 ubuntu 20.04 droplet and everything works…

Your help is much appreciated as I have been stuck for the last 8 hours..

Answer 1

andreacabral2 February 3, 2021

Hey asb,
indeed adding the rule to allow dns traffic fixed the pinging issue but not the installation of a package through apt. So now doing:

$ ping -c 1 google.com

works and is able to send and receive the packet.
And of course since apt uses mostly http and for docker it uses https I needed to add those outbound rules as well. Finally it seems I also needed icmp, not entirely sure why.
In any case here is the set of outbound rules I needed to get nginx installed when having a DO cloud firewall:

  outbound_rule {
    protocol = "tcp"
    port_range = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "tcp"
    port_range = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "tcp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "udp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "icmp"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

Thanks for the help asb!

Reply Report

Answer 2

andreacabral2 February 3, 2021

Hey thanks for getting back to me. Unfortunately that did not work and the last command still returns:

$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nginx

Reply Report

Answer 3

andreacabral2 February 3, 2021

Also I have an errata to add to my description, maybe that helps. Obviously when I explained that I ping'ed an IP directly with

$ ping -c 1 4.2.2.1

I did not get back a response from google.com?! I might have copy-pasted that output from a command I ran on another server to compare by mistake.
In any case, the result from pinging a direct IP on this faulty server was and still is:

$ ping -c 1 4.2.2.1
PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

--- 4.2.2.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Seems like I can send packets, but not receive?

Reply Report

Answer 4

andreacabral2 February 3, 2021

I was able to track down the issue to indeed the digitalocean firewall added with terraform. I applied the whole infra minus the firewall and nginx was able to be installed and was running fine.

Not sure what could be wrong with the following firewall though:

resource "digitalocean_firewall" "apos-algo-backend" {
  name = "only-apos-algo-backend"

  droplet_ids = [digitalocean_droplet.www-1.id]

  # inbound_rule {
  #   protocol         = "tcp"
  #   port_range       = "80"
  #   source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
  # }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "80"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "22"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "tcp"
    port_range            = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "udp"
    port_range            = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
}

Any idea well appreciated, thanks!

Reply Report

Related

Question

Why is my droplet not installing nginx through terraform and cloud init

Related

Leave a comment

Looking for something else?

Sign up for our newsletter

Related

Question

Why is my droplet not installing nginx through terraform and cloud init

Related

Leave a comment

Related

question

Serving two websites on one Nginx

question

Connect to DB installed into dedicated VPS (private networking)

question

Perfect way to install nginx with pagespeed on Ubuntu

question

I have a few question about moint configuration

Looking for something else?