I’m using terraform to provision droplets (which I’ve done before without issue), and today I ran into a problem I haven’t seen before. Droplets are unable to reach the outside network
at all. At first I thought it was a DNS problem, but literally every public IP
is inaccessible from within these droplets. I’ve reproduced this on several
images and I’m at my wits end about how to solve this.

Everything else is working smoothly, I can reach the droplet via SSH without issue
after creation, but I cannot reach any outside hostname or IP address.

Here’s what my internal network looks like:

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group
default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAUL
T group default qlen 1000
    link/ether b2:fc:48:21:99:b5 brd ff:ff:ff:ff:ff:ff
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode
DEFAULT group default
    link/ether 02:42:3b:8f:31:6c brd ff:ff:ff:ff:ff:ff

Here is the relevant section of my terraform config:

resource "digitalocean_droplet" "staging-cafe" {
  image = "docker-20-04"
  name = "staging-cafe"
  region = "sfo2"
  size = "s-2vcpu-4gb"
  private_networking = false
  monitoring = true
  ssh_keys = [
    digitalocean_ssh_key.hal.fingerprint
  ]
}

resource "digitalocean_firewall" "cafe" {
  name = "cafe"
  droplet_ids = [ digitalocean_droplet.staging-cafe.id ]
  inbound_rule {
      protocol = "tcp"
      port_range = "22"
      source_addresses = var.vpn_source_addrs
  }
  inbound_rule {
      protocol = "tcp"
      port_range = "443"
  }
  inbound_rule {
    protocol = "icmp"
  }
  outbound_rule {
    protocol = "tcp"
    port_range = "53"
  }
  outbound_rule {
    protocol = "udp"
    port_range = "53"
  }
  outbound_rule {
    protocol = "icmp"
  }
}

Again, this all works fine! But the host has no outgoing network access, only
incoming. What other information is needed to troubleshoot this issue?

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hey @zenhob -

Firstly, thank you for writing such a clear and complete question. That is helpful.

I think this comes down to your firewall rules, and some gotchas related to how Cloud Firewalls behave, and how they’re configured via Terraform.

Context: Cloud Firewalls start completely locked down, and the rules are additive. Because of this, it is common to set OUTBOUND rules that are very permissive. For example, here is the default outbound firewall config when done via UI. That same outbound rule in terraform would look like this:

  outbound_rule {
    protocol = "icmp"
  }
  outbound_rule {
    protocol = "tcp"
    port_range = "1-65535"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
  outbound_rule {
    protocol = "udp"
    port_range = "1-65535"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

Problem You need to add the destination_addresses attribute to your outbound rules. I’m surprised Terraform even allows you to set an outbound rule without a port range, effectively what that is doing is leaving it blocked. See terraform docs for firewall rules

So you need to update it to say, “allow connection to any IP”

outbound_rule {
    protocol = "tcp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }
  outbound_rule {
    protocol = "udp"
    port_range = "53"
    destination_addresses = ["0.0.0.0/0", "::/0"]
  }

or you could use some kind of specific range of dns servers.

Once you’ve made that change, you can communicate outbound ONLY VIA port 53. Which is still pretty restrictive.

For example that will allow you to do a dns lookup like dig @8.8.8.8 www.google.com, but a curl www.google.com will still fail because HTTP and HTTPS outbound ports are still blocked.

Hope that helps,
Andy

Submit an Answer