I am provisioning the infrastructure using terraform to get a droplet and a pg db cluster up and running. The droplet frontend is nginx and it’s installed through the user-data providing a cloud-config file. However, it fails to install nginx and seems to not be connected to the internet at all. I don’t really understand as I tried the same cloud config file alone by using the cloud digitalocean dashboard and copy pasting my cloud config file into the user data form item when creating a droplet and it worked. Am I missing something?
My cloud config file
#cloud-config
package_update: true
package_upgrade: false
package_reboot_if_required: false
packages:
- nginx
write_files:
- content: |
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
server {
listen 80 proxy_protocol;
server_name assessment-test.gaitup.com cloud.digitalocean.com;
location / {
proxy_pass http://localhost:${PORT};
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
proxy_set_header Host $host;
}
}
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
log_format main '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
path: /etc/nginx/nginx.conf
- content: |
PORT=${PORT}
DATABASE_URL=${DATABASE_URL}
path: /root/.env
runcmd:
- ufw allow 'Nginx HTTP'
- ufw allow 'Nginx HTTPS'
- systemctl restart nginx
my terraform file for the drople:
resource "digitalocean_droplet" "www-1" {
image = "docker-20-04"
name = format("%s.com.gaitup.%s.www-1", var.env, var.app_name)
region = var.region
size = "s-1vcpu-1gb"
# private_networking = true
vpc_uuid = data.digitalocean_vpc.www-vpc.id
backups = true
monitoring = true
ssh_keys = [
data.digitalocean_ssh_key.terraform_do_rsa.id
]
connection {
host = self.ipv4_address
user = "root"
type = "ssh"
private_key = file(pathexpand("~/${var.pvt_key}"))
timeout = "2m"
}
user_data = templatefile("${path.root}/www-cloud-init.yaml", {
APP_VERSION = var.app_version,
APP_NAME = var.app_name,
DO_API_PERSONAL_TOKEN = var.do_token,
PORT = var.port,
DATABASE_URL = format("%s:%d/%s", digitalocean_database_cluster.www-db.private_host, digitalocean_database_cluster.www-db.port, digitalocean_database_db.gaitup_db.name)
})
}
resource "digitalocean_database_firewall" "www-fw" {
cluster_id = digitalocean_database_cluster.www-db.id
rule {
type = "droplet"
value = digitalocean_droplet.www-1.id
}
}
resource "digitalocean_database_db" "gaitup_db" {
cluster_id = digitalocean_database_cluster.www-db.id
name = format("gaitup_%s", var.env)
}
resource "digitalocean_database_cluster" "www-db" {
name = format("%s-com-gaitup-%s-db", var.env, replace(var.app_name, "/[_\\.]/g", "-"))
engine = "pg"
version = "12"
size = "db-s-1vcpu-1gb"
region = var.region
node_count = 1
private_network_uuid = data.digitalocean_vpc.www-vpc.id
}
resource "digitalocean_firewall" "apos-algo-backend" {
name = "only-apos-algo-backend"
droplet_ids = [digitalocean_droplet.www-1.id]
# inbound_rule {
# protocol = "tcp"
# port_range = "80"
# source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
# }
inbound_rule {
protocol = "tcp"
port_range = "80"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "443"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "22"
source_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "80"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "443"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
}
Now what I have been able to see looking at the log files of cloud init is the following:
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:config' at Tue, 02 Feb 2021 17:00:07 +0000. Up 18.83 seconds.
Err:1 https://download.docker.com/linux/ubuntu focal InRelease
Temporary failure resolving 'download.docker.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease Temporary failure resolving 'download.docker.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package nginx
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:final' at Tue, 02 Feb 2021 17:00:28 +0000. Up 40.53 seconds.
2021-02-02 17:01:20,233 - util.py[WARNING]: Failed to install packages: ['nginx']
2021-02-02 17:01:20,392 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2021-02-02 17:01:20,394 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
curl: (6) Could not resolve host: insights.nyc3.cdn.digitaloceanspaces.com
Initializing machine ID from D-Bus machine ID.
ERROR: Could not find a profile matching 'Nginx HTTP'
ERROR: Could not find a profile matching 'Nginx HTTPS'
Failed to restart nginx.service: Unit nginx.service not found.
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Unable to find image 'registry.digitalocean.com/gaitup/gaitup-live-customer-licensing:0.0.1' locally
docker: Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution.
See 'docker run --help'.
2021-02-02 17:03:51,407 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2021-02-02 17:03:51,408 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 finished at Tue, 02 Feb 2021 17:03:51 +0000. Datasource DataSourceDigitalOcean. Up 243.16 seconds
It does not seem to have any internet connection. I was able to confirm that by doing in the droplet
$ ping -c 1 google.com # -> it fails with "ping: google.com: Temporary failure in name resolution"
However pinging an IP directly:
$ ping -c 1 4.2.2.1
returns
PING google.com (172.217.168.46): 56 data bytes
64 bytes from 172.217.168.46: icmp_seq=0 ttl=57 time=12.742 ms
--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.742/12.742/12.742/0.000 ms
So some issue maybe with DNS maybe? Trying to nslookup google indeed fails:
$ nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53
** server can't find google.com: SERVFAIL
So I thought to look at the firewall with ufw
$ ufw status
22/tcp LIMIT Anywhere
2375/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
22/tcp (v6) LIMIT Anywhere (v6)
2375/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
Looks like traffic on port 80 and 443 is not allowed. So next move was to allow it:
$ ufw allow HTTP
$ ufw allow HTTPS
$ ufw status
22/tcp LIMIT Anywhere
2375/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
80 ALLOW Anywhere
443 ALLOW Anywhere
22/tcp (v6) LIMIT Anywhere (v6)
2375/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
now it they are allowed. Next I re try a pinging:
$ ping -c 1 google.com
ping: google.com: Temporary failure in name resolution
Last, since the provisioning using cloud-config did not work. I thought at this point let’s give it a try manually. So I ssh into the droplet and start with:
$ sudo apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 https://download.docker.com/linux/ubuntu focal InRelease
Temporary failure resolving 'download.docker.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease Temporary failure resolving 'download.docker.com'
Nope. Nor does:
$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
Now I am stuck. I also should mention, on another server, using the same approach, with the same cloud-config file, the droplet was able to install nginx in docker 18 ubuntu 20.04 droplet and everything works…
Your help is much appreciated as I have been stuck for the last 8 hours…
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
Join our DigitalOcean community of over a million developers for free! Get help and share knowledge in Q&A, subscribe to topics of interest, and get courses and tools that will help you grow as a developer and scale your project or business.
Hey asb, indeed adding the rule to allow dns traffic fixed the pinging issue but not the installation of a package through apt. So now doing:
works and is able to send and receive the packet. And of course since apt uses mostly http and for docker it uses https I needed to add those outbound rules as well. Finally it seems I also needed icmp, not entirely sure why. In any case here is the set of outbound rules I needed to get nginx installed when having a DO cloud firewall:
Thanks for the help asb!
I was able to track down the issue to indeed the digitalocean firewall added with terraform. I applied the whole infra minus the firewall and nginx was able to be installed and was running fine.
Not sure what could be wrong with the following firewall though:
Any idea well appreciated, thanks!
Also I have an errata to add to my description, maybe that helps. Obviously when I explained that I ping’ed an IP directly with
I did not get back a response from google.com?! I might have copy-pasted that output from a command I ran on another server to compare by mistake. In any case, the result from pinging a direct IP on this faulty server was and still is:
Seems like I can send packets, but not receive?