I am provisioning the infrastructure using terraform to get a droplet and a pg db cluster up and running. The droplet frontend is nginx and it’s installed through the user-data providing a cloud-config file. However, it fails to install nginx and seems to not be connected to the internet at all. I don’t really understand as I tried the same cloud config file alone by using the cloud digitalocean dashboard and copy pasting my cloud config file into the user data form item when creating a droplet and it worked. Am I missing something?
My cloud config file
#cloud-config
package_update: true
package_upgrade: false
package_reboot_if_required: false
packages:
- nginx
write_files:
- content: |
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 768;
}
http {
server {
listen 80 proxy_protocol;
server_name assessment-test.gaitup.com cloud.digitalocean.com;
location / {
proxy_pass http://localhost:${PORT};
proxy_set_header X-Real-IP $proxy_protocol_addr;
proxy_set_header X-Forwarded-For $proxy_protocol_addr;
proxy_set_header Host $host;
}
}
##
# Basic Settings
##
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
include /etc/nginx/mime.types;
default_type application/octet-stream;
##
# Logging Settings
##
log_format main '$proxy_protocol_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
error_log /var/log/nginx/error.log;
##
# Gzip Settings
##
gzip on;
##
# Virtual Host Configs
##
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
path: /etc/nginx/nginx.conf
- content: |
PORT=${PORT}
DATABASE_URL=${DATABASE_URL}
path: /root/.env
runcmd:
- ufw allow 'Nginx HTTP'
- ufw allow 'Nginx HTTPS'
- systemctl restart nginx
my terraform file for the drople:
resource "digitalocean_droplet" "www-1" {
image = "docker-20-04"
name = format("%s.com.gaitup.%s.www-1", var.env, var.app_name)
region = var.region
size = "s-1vcpu-1gb"
# private_networking = true
vpc_uuid = data.digitalocean_vpc.www-vpc.id
backups = true
monitoring = true
ssh_keys = [
data.digitalocean_ssh_key.terraform_do_rsa.id
]
connection {
host = self.ipv4_address
user = "root"
type = "ssh"
private_key = file(pathexpand("~/${var.pvt_key}"))
timeout = "2m"
}
user_data = templatefile("${path.root}/www-cloud-init.yaml", {
APP_VERSION = var.app_version,
APP_NAME = var.app_name,
DO_API_PERSONAL_TOKEN = var.do_token,
PORT = var.port,
DATABASE_URL = format("%s:%d/%s", digitalocean_database_cluster.www-db.private_host, digitalocean_database_cluster.www-db.port, digitalocean_database_db.gaitup_db.name)
})
}
resource "digitalocean_database_firewall" "www-fw" {
cluster_id = digitalocean_database_cluster.www-db.id
rule {
type = "droplet"
value = digitalocean_droplet.www-1.id
}
}
resource "digitalocean_database_db" "gaitup_db" {
cluster_id = digitalocean_database_cluster.www-db.id
name = format("gaitup_%s", var.env)
}
resource "digitalocean_database_cluster" "www-db" {
name = format("%s-com-gaitup-%s-db", var.env, replace(var.app_name, "/[_\\.]/g", "-"))
engine = "pg"
version = "12"
size = "db-s-1vcpu-1gb"
region = var.region
node_count = 1
private_network_uuid = data.digitalocean_vpc.www-vpc.id
}
resource "digitalocean_firewall" "apos-algo-backend" {
name = "only-apos-algo-backend"
droplet_ids = [digitalocean_droplet.www-1.id]
# inbound_rule {
# protocol = "tcp"
# port_range = "80"
# source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
# }
inbound_rule {
protocol = "tcp"
port_range = "80"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "443"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "22"
source_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "80"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "443"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
}
Now what I have been able to see looking at the log files of cloud init is the following:
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:config' at Tue, 02 Feb 2021 17:00:07 +0000. Up 18.83 seconds.
Err:1 https://download.docker.com/linux/ubuntu focal InRelease
Temporary failure resolving 'download.docker.com'
Err:2 http://archive.ubuntu.com/ubuntu focal InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:3 http://security.ubuntu.com/ubuntu focal-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists...
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease Temporary failure resolving 'download.docker.com'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
Calculating upgrade...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package nginx
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 running 'modules:final' at Tue, 02 Feb 2021 17:00:28 +0000. Up 40.53 seconds.
2021-02-02 17:01:20,233 - util.py[WARNING]: Failed to install packages: ['nginx']
2021-02-02 17:01:20,392 - cc_package_update_upgrade_install.py[WARNING]: 1 failed with exceptions, re-raising the last one
2021-02-02 17:01:20,394 - util.py[WARNING]: Running module package-update-upgrade-install (<module 'cloudinit.config.cc_package_update_upgrade_install' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_package_update_upgrade_install.py'>) failed
curl: (6) Could not resolve host: insights.nyc3.cdn.digitaloceanspaces.com
Initializing machine ID from D-Bus machine ID.
ERROR: Could not find a profile matching 'Nginx HTTP'
ERROR: Could not find a profile matching 'Nginx HTTPS'
Failed to restart nginx.service: Unit nginx.service not found.
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution
Unable to find image 'registry.digitalocean.com/gaitup/gaitup-live-customer-licensing:0.0.1' locally
docker: Error response from daemon: Get https://registry.digitalocean.com/v2/: dial tcp: lookup registry.digitalocean.com: Temporary failure in name resolution.
See 'docker run --help'.
2021-02-02 17:03:51,407 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
2021-02-02 17:03:51,408 - util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_scripts_user.py'>) failed
Cloud-init v. 20.3-2-g371b392c-0ubuntu1~20.04.1 finished at Tue, 02 Feb 2021 17:03:51 +0000. Datasource DataSourceDigitalOcean. Up 243.16 seconds
It does not seem to have any internet connection. I was able to confirm that by doing in the droplet
$ ping -c 1 google.com # -> it fails with "ping: google.com: Temporary failure in name resolution"
However pinging an IP directly:
$ ping -c 1 4.2.2.1
returns
PING google.com (172.217.168.46): 56 data bytes
64 bytes from 172.217.168.46: icmp_seq=0 ttl=57 time=12.742 ms
--- google.com ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.742/12.742/12.742/0.000 ms
So some issue maybe with DNS maybe? Trying to nslookup google indeed fails:
$ nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53#53
** server can't find google.com: SERVFAIL
So I thought to look at the firewall with ufw
$ ufw status
22/tcp LIMIT Anywhere
2375/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
22/tcp (v6) LIMIT Anywhere (v6)
2375/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
Looks like traffic on port 80 and 443 is not allowed. So next move was to allow it:
$ ufw allow HTTP
$ ufw allow HTTPS
$ ufw status
22/tcp LIMIT Anywhere
2375/tcp ALLOW Anywhere
2376/tcp ALLOW Anywhere
80 ALLOW Anywhere
443 ALLOW Anywhere
22/tcp (v6) LIMIT Anywhere (v6)
2375/tcp (v6) ALLOW Anywhere (v6)
2376/tcp (v6) ALLOW Anywhere (v6)
80 (v6) ALLOW Anywhere (v6)
443 (v6) ALLOW Anywhere (v6)
now it they are allowed. Next I re try a pinging:
$ ping -c 1 google.com
ping: google.com: Temporary failure in name resolution
Last, since the provisioning using cloud-config did not work. I thought at this point let’s give it a try manually. So I ssh into the droplet and start with:
$ sudo apt update
Err:1 http://archive.ubuntu.com/ubuntu focal InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:2 http://security.ubuntu.com/ubuntu focal-security InRelease
Temporary failure resolving 'security.ubuntu.com'
Err:3 https://download.docker.com/linux/ubuntu focal InRelease
Temporary failure resolving 'download.docker.com'
Err:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease
Temporary failure resolving 'archive.ubuntu.com'
Err:5 http://archive.ubuntu.com/ubuntu focal-backports InRelease
Temporary failure resolving 'archive.ubuntu.com'
Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving 'archive.ubuntu.com'
W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving 'security.ubuntu.com'
W: Failed to fetch https://download.docker.com/linux/ubuntu/dists/focal/InRelease Temporary failure resolving 'download.docker.com'
Nope. Nor does:
$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
Now I am stuck. I also should mention, on another server, using the same approach, with the same cloud-config file, the droplet was able to install nginx in docker 18 ubuntu 20.04 droplet and everything works…
Your help is much appreciated as I have been stuck for the last 8 hours…
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Hey thanks for getting back to me. Unfortunately that did not work and the last command still returns:
$ sudo apt install nginx
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package nginx
Also I have an errata to add to my description, maybe that helps. Obviously when I explained that I ping’ed an IP directly with
$ ping -c 1 4.2.2.1
I did not get back a response from google.com?! I might have copy-pasted that output from a command I ran on another server to compare by mistake. In any case, the result from pinging a direct IP on this faulty server was and still is:
$ ping -c 1 4.2.2.1
PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.
--- 4.2.2.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms
Seems like I can send packets, but not receive?
I was able to track down the issue to indeed the digitalocean firewall added with terraform. I applied the whole infra minus the firewall and nginx was able to be installed and was running fine.
Not sure what could be wrong with the following firewall though:
resource "digitalocean_firewall" "apos-algo-backend" {
name = "only-apos-algo-backend"
droplet_ids = [digitalocean_droplet.www-1.id]
# inbound_rule {
# protocol = "tcp"
# port_range = "80"
# source_addresses = [data.aws_eip.gaitup-live-customer-apos-algo-backend.public_ip]
# }
inbound_rule {
protocol = "tcp"
port_range = "80"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "443"
source_addresses = ["0.0.0.0/0", "::/0"]
}
inbound_rule {
protocol = "tcp"
port_range = "22"
source_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "80"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "tcp"
port_range = "443"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
outbound_rule {
protocol = "udp"
port_range = "53"
destination_addresses = ["0.0.0.0/0", "::/0"]
}
}
Any idea well appreciated, thanks!
Become a contributor for community
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.
DigitalOcean Documentation
Full documentation for every DigitalOcean product.
Resources for startups and SMBs
The Wave has everything you need to know about building a business, from raising funding to marketing your product.
The developer cloud
Scale up as you grow — whether you're running one virtual machine or ten thousand.
Get started for free
Sign up and get $200 in credit for your first 60 days with DigitalOcean.*
*This promotional offer applies to new accounts only.