DigitalOcean Kubernetes: new control plane is faster and free, enable HA for 99.95% uptime SLA

Question

Why is my droplet not installing nginx through terraform and cloud init

I am provisioning the infrastructure using terraform to get a droplet and a pg db cluster up and running. The droplet frontend is nginx and it’s installed through the user-data providing a cloud-config file. However, it fails to install nginx and seems to not be connected to the internet at all. I don’t really understand as I tried the same cloud config file alone by using the cloud digitalocean dashboard and copy pasting my cloud config file into the user data form item when creating a droplet and it worked. Am I missing something?

My cloud config file

#cloud-config

package_update: true
package_upgrade: false
package_reboot_if_required: false

packages:
  - nginx

write_files:
  - content: |
      user www-data;
      worker_processes auto;
      pid /run/nginx.pid;
      include /etc/nginx/modules-enabled/*.conf;

      events {
        worker_connections 768;
      }

      http {
        server {
          listen	80	proxy_protocol;
          server_name	assessment-test.gaitup.com cloud.digitalocean.com;
          location / {
            proxy_pass       http://localhost:${PORT};
            proxy_set_header X-Real-IP       $proxy_protocol_addr;
            proxy_set_header X-Forwarded-For $proxy_protocol_addr;
            proxy_set_header Host $host;
          }
        }

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Logging Settings
        ##

        log_format main '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$http_x_forwarded_for"';

        access_log /var/log/nginx/access.log main;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
      }
    path: /etc/nginx/nginx.conf
  - content: |
      PORT=${PORT}
      DATABASE_URL=${DATABASE_URL}
    path: /root/.env

runcmd:
- ufw allow 'Nginx HTTP'
- ufw allow 'Nginx HTTPS'
- systemctl restart nginx

my terraform file for the drople:

resource "digitalocean_droplet" "www-1" {
  image              = "docker-20-04"
  name               = format("%s.com.gaitup.%s.www-1", var.env, var.app_name)
  region             = var.region
  size               = "s-1vcpu-1gb"
  # private_networking = true
  vpc_uuid           = data.digitalocean_vpc.www-vpc.id
  backups            = true
  monitoring         = true
  ssh_keys = [
    data.digitalocean_ssh_key.terraform_do_rsa.id
  ]
  connection {
    host        = self.ipv4_address
    user        = "root"
    type        = "ssh"
    private_key = file(pathexpand("~/${var.pvt_key}"))
    timeout     = "2m"
  }
  user_data = templatefile("${path.root}/www-cloud-init.yaml", { 
    APP_VERSION = var.app_version, 
    APP_NAME = var.app_name,
    DO_API_PERSONAL_TOKEN = var.do_token,
    PORT                  = var.port,
    DATABASE_URL          = format("%s:%d/%s", digitalocean_database_cluster.www-db.private_host, digitalocean_database_cluster.www-db.port, digitalocean_database_db.gaitup_db.name)
  })
}

resource "digitalocean_database_firewall" "www-fw" {
  cluster_id = digitalocean_database_cluster.www-db.id

  rule {
    type  = "droplet"
    value = digitalocean_droplet.www-1.id
  }
}

resource "digitalocean_database_db" "gaitup_db" {
  cluster_id = digitalocean_database_cluster.www-db.id
  name       = format("gaitup_%s", var.env)
}

resource "digitalocean_database_cluster" "www-db" {
  name                 = format("%s-com-gaitup-%s-db", var.env, replace(var.app_name, "/[_\\.]/g", "-"))
  engine               = "pg"
  version              = "12"
  size                 = "db-s-1vcpu-1gb"
  region               = var.region
  node_count           = 1
  private_network_uuid = data.digitalocean_vpc.www-vpc.id
}

resource "digitalocean_firewall" "apos-algo-backend" {
  name = "only-apos-algo-backend"

  droplet_ids = [digitalocean_droplet.www-1.id]

  # inbound_rule {
  #   protocol         = "tcp"
  #   port_range       = "80"
  #   source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
  # }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "80"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "22"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "tcp"
    port_range            = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
}

Now what I have been able to see looking at the log files of cloud init is the following:

Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:config' at Tue, 02 Feb 2021 17:00:07 +0000. Up 18.83 seconds.
Err:1 https://download.docker.com/linux/ubuntu focal InRelease
  Temporary failure resolving 'download.docker.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease  Temporary failure resolving 'download.docker.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package nginx
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:final' at Tue, 02 Feb 2021 17:00:28 +0000. Up 40.53 seconds.
2021-02-02 17:01:20,233 - util.py[WARNING]: Failed to install packages: ['nginx']
2021-02-02 17:01:20,392 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2021-02-02 17:01:20,394 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
curl: (6) Could not resolve host: insights.nyc3.cdn.digitaloceanspaces.com
Initializing machine ID from D-Bus machine ID.
ERROR: Could not find a profile matching 'Nginx HTTP'
ERROR: Could not find a profile matching 'Nginx HTTPS'
Failed to restart nginx.service: Unit nginx.service not found.
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Unable to find image 'registry.digitalocean.com/gaitup/gaitup-live-customer-licensing:0.0.1' locally
docker: Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution.
See 'docker run --help'.
2021-02-02 17:03:51,407 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2021-02-02 17:03:51,408 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 finished at Tue, 02 Feb 2021 17:03:51 +0000. Datasource DataSourceDigitalOcean.  Up 243.16 seconds

It does not seem to have any internet connection. I was able to confirm that by doing in the droplet

$ ping -c 1 google.com # -> it fails with "ping: google.com: Temporary failure in name resolution"

However pinging an IP directly:

$ ping -c 1 4.2.2.1

returns

PING google.com (172.217.168.46): 56 data bytes
64 bytes from 172.217.168.46: icmp_seq=0 ttl=57 time=12.742 ms

--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.742/12.742/12.742/0.000 ms

So some issue maybe with DNS maybe? Trying to nslookup google indeed fails:

$ nslookup google.com
Server:		127.0.0.53
Address:	127.0.0.53#53

** server can't find google.com: SERVFAIL

So I thought to look at the firewall with ufw

$ ufw status
22/tcp                     LIMIT       Anywhere
2375/tcp                   ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
22/tcp (v6)                LIMIT       Anywhere (v6)
2375/tcp (v6)              ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)

Looks like traffic on port 80 and 443 is not allowed. So next move was to allow it:

$ ufw allow HTTP
$ ufw allow HTTPS
$ ufw status
22/tcp                     LIMIT       Anywhere
2375/tcp                   ALLOW       Anywhere
2376/tcp                   ALLOW       Anywhere
80                         ALLOW       Anywhere
443                        ALLOW       Anywhere
22/tcp (v6)                LIMIT       Anywhere (v6)
2375/tcp (v6)              ALLOW       Anywhere (v6)
2376/tcp (v6)              ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
443 (v6)                   ALLOW       Anywhere (v6)

now it they are allowed. Next I re try a pinging:

$ ping -c 1 google.com
ping: google.com: Temporary failure in name resolution

Last, since the provisioning using cloud-config did not work. I thought at this point let’s give it a try manually. So I ssh into the droplet and start with:

$ sudo apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
  Temporary failure resolving 'security.ubuntu.com'
Err:3 https://download.docker.com/linux/ubuntu focal InRelease
  Temporary failure resolving 'download.docker.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
  Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease  Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease  Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease  Temporary failure resolving 'download.docker.com'

Nope. Nor does:

$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done

Now I am stuck. I also should mention, on another server, using the same approach, with the same cloud-config file, the droplet was able to install nginx in docker 18 ubuntu 20.04 droplet and everything works…

Your help is much appreciated as I have been stuck for the last 8 hours…

Show comments
Submit an answer

This textbox defaults to using Markdown to format your answer.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Sign In or Sign Up to Answer

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Want to learn more? Join the DigitalOcean Community!

Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.

Sign up now

andreacabral2 • February 3, 2021

Hey asb, indeed adding the rule to allow dns traffic fixed the pinging issue but not the installation of a package through apt. So now doing:

$ ping -c 1 google.com

works and is able to send and receive the packet. And of course since apt uses mostly http and for docker it uses https I needed to add those outbound rules as well. Finally it seems I also needed icmp, not entirely sure why. In any case here is the set of outbound rules I needed to get nginx installed when having a DO cloud firewall:

  outbound_rule {
    protocol = "tcp"
    port_range = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "tcp"
    port_range = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "tcp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "udp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol = "icmp"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

Thanks for the help asb!

andreacabral2 • February 3, 2021

I was able to track down the issue to indeed the digitalocean firewall added with terraform. I applied the whole infra minus the firewall and nginx was able to be installed and was running fine.

Not sure what could be wrong with the following firewall though:

resource "digitalocean_firewall" "apos-algo-backend" {
  name = "only-apos-algo-backend"

  droplet_ids = [digitalocean_droplet.www-1.id]

  # inbound_rule {
  #   protocol         = "tcp"
  #   port_range       = "80"
  #   source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
  # }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "80"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  inbound_rule {
    protocol         = "tcp"
    port_range       = "22"
    source_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "tcp"
    port_range            = "80"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol         = "tcp"
    port_range       = "443"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

  outbound_rule {
    protocol              = "udp"
    port_range            = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
}

Any idea well appreciated, thanks!

andreacabral2 • February 3, 2021

Also I have an errata to add to my description, maybe that helps. Obviously when I explained that I ping’ed an IP directly with

$ ping -c 1 4.2.2.1

I did not get back a response from google.com?! I might have copy-pasted that output from a command I ran on another server to compare by mistake. In any case, the result from pinging a direct IP on this faulty server was and still is:

$ ping -c 1 4.2.2.1
PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

--- 4.2.2.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Seems like I can send packets, but not receive?

Try DigitalOcean for free

Click below to sign up and get $100 of credit to try our products over 60 days!

Sign up