We hope you find this tutorial helpful. In addition to guides like this one, we provide simple cloud infrastructure for developers. Learn more →

How To Troubleshoot DigitalOcean Firewalls

PostedJune 6, 2017 1.8k views Firewall DigitalOcean

Introduction

DigitalOcean Cloud Firewalls improve the security of your infrastructure by evaluating traffic to your Droplets against a set of configurable rules. While DigitalOcean designed Firewalls to be intuitive, the combination of multiple network policies and interaction with software active on Droplets can sometimes lead to unexpected results.

In this guide, we will discuss strategies to gather information about your network policies and troubleshoot DigitalOcean Cloud Firewall issues. We will cover how to find your active Firewalls and how to view the comprehensive set of rules applied to a single Droplet. We will also discuss how to discover host-based firewalls that might be active on Droplets and how to mitigate or resolve conflicting policies.

Finding Firewall Information in the DigitalOcean Control Panel

The first step in diagnosing problems in your Firewall policy is to understand the which rules are being applied. The DigitalOcean Control Panel contains information about individual Firewalls as well as the combined ruleset that applies to each Droplet.

Displaying Information About All Firewalls

To view all of your Firewalls in the DigitalOcean Control Panel, click on the Networking item in the main navigation pane and then select the Firewalls subitem:

DigitalOcean navigate to firewalls

The Firewall index page lists basic information about each of your Firewalls like the number of Droplets it applies to, the number of rules it contains, and when it was created:

DigitalOcean Firewalls index

Clicking the name of a Firewall takes you to the Firewall detail page where you can see the policy's individual rules and the Droplets it targets:

DigitalOcean individual firewall page

Reviewing the rules applied by each of your Firewalls can provide an overview of the types of network policy being enforced on your infrastructure as a whole.

Displaying the Firewall Policy Affecting an Individual Droplet

The Firewall index and detail pages are not the best place to look if you are interested in the policy affecting an individual Droplet. Instead, check the Droplet's detail page.

From the Droplets item in the main Control Panel navigation menu, click the Droplet's name. From there, click Networking in the Droplet menu:

DigitalOcean Droplet menu

Towards the bottom of the page, a section called Firewalls lists the Firewalls that target the Droplet. Below, the policies from each of the Droplet's Firewalls are combined into a comprehensive table:

DigitalOcean combined ruleset

These inbound and outbound rules represent all of the traffic allowed through the Firewall for the Droplet. If you are experiencing issues with connectivity to your Droplet after applying a Firewall policy, this is a good place to look.

Displaying Host-Based Firewall Rules Active on Droplets

While the DigitalOcean Cloud Firewall service provides a network-based firewall solution, many administrators may also have a host-based firewall on the Droplet itself. Popular host-based firewalls include IPTables, UFW, and firewalld. When troubleshooting connectivity issues, it is important to check these services that may also be filtering traffic on the Droplet level.

Finding the UFW Firewall Settings

If your server runs a UFW firewall, you can view the current filtering rules by typing:

  • sudo ufw status verbose
Output
Status: active Logging: on (low) Default: deny (incoming), allow (outgoing), disabled (routed) New profiles: skip To Action From -- ------ ---- 22/tcp (OpenSSH) ALLOW IN Anywhere 22/tcp (OpenSSH (v6)) ALLOW IN Anywhere (v6)

The output shows a few things:

  • The Status line indicates that the firewall is actively enforcing traffic rules.
  • The Default line indicates that the incoming policy is to deny traffic and the outgoing policy is to allow traffic.
  • The ruleset at the bottom indicates that incoming connections to port 22 are allowed on both IPv4 and IPv6 addresses.

If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your UFW firewall by typing:

  • sudo ufw disable

If you want to keep UFW in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the UFW rules by following the UFW Essentials: Common Firewall Rules and Commands guide.

Finding the Firewalld Firewall Settings

If your Droplet runs a firewalld firewall, you can view the current traffic rules with a sequence of checks.

First, check whether firewalld is active:

  • sudo firewall-cmd --state
Output
running

If the firewall is running, check the active zones:

  • sudo firewall-cmd --get-active-zones
Output
public interfaces: eth0

If firewalld has active zones, it means that it is evaluating traffic against a set of rules. We can display the ports and services associated with each of the active zones with the --info-zone option:

  • sudo firewall-cmd --info-zone=public
Output
public (active) target: default icmp-block-inversion: no interfaces: eth0 sources: services: dhcpv6-client ssh ports: protocols: masquerade: no forward-ports: sourceports: icmp-blocks: rich rules:

Check whether any ports are open by looking at:

  • The target value: If this is set to DROP or %%REJECT%%, all traffic will be denied regardless of the other settings.
  • The ports value: this lists ports explicitly allowed through the firewall
  • The services value: this lists services that are allowed through. These are specific sets rules that can be allowed through as a unit.

Here, we can see that the dhcpv6-client and ssh services are allowed through the firewall. We can check the ports associated with these services by typing:

  • sudo firewall-cmd --permanent --get-ports --service=dhcpv6-client
  • sudo firewall-cmd --permanent --get-ports --service=ssh
Output
546/udp 22/tcp

If you have a DigitalOcean Cloud Firewall in place with conflicting rules, you can disable your firewalld firewall by typing:

  • sudo systemctl stop firewalld
  • sudo systemctl disable firewalld

If you want to keep firewalld in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the firewalld rules by following the How To Set Up a Firewall Using FirewallD on CentOS 7 guide.

Finding the IPTables Firewall Settings

If you are using an IPTables firewall, you can view the current IPv4 filtering rules by typing:

  • sudo iptables --line-numbers -vL
Output
Chain INPUT (policy DROP 1 packets, 40 bytes) num pkts bytes target prot opt in out source destination 1 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 2 764 56512 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 3 9 540 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 4 0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination Chain OUTPUT (policy ACCEPT 374 packets, 97673 bytes) num pkts bytes target prot opt in out source destination

The output indicates:

  • The default policy for incoming and outgoing traffic (DROP for incoming, and ACCEPT for outgoing)
  • The exceptions to the default policies. Here, we can see exemptions for incoming TCP traffic to ports 22 and 80 (as well as traffic related to existing connections and traffic coming from the local interface)

If you are more comfortable with IPTables configuration syntax, you may find the rules-based view more helpful:

  • sudo iptables -S
Output
-P INPUT DROP -P FORWARD ACCEPT -P OUTPUT ACCEPT -A INPUT -i lo -j ACCEPT -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT

The iptables command only displays the rules for filtering IPv4 traffic. To show the IPv6 filtering rules, rerun the above commands using the ip6tables instead of iptables.

If your DigitalOcean Cloud Firewall has rules that conflict with your IPTables firewall, you can disable your IPTables firewall by typing:

  • sudo iptables -P INPUT ACCEPT
  • sudo iptables -P OUTPUT ACCEPT
  • sudo iptables -P FORWARD ACCEPT
  • sudo iptables -F
  • sudo ip6tables -P INPUT ACCEPT
  • sudo ip6tables -P OUTPUT ACCEPT
  • sudo ip6tables -P FORWARD ACCEPT
  • sudo ip6tables -F

If you are using a service like iptables-persistent or have a script loading IPTables rules at boot, you may have to disable those as well.

If you want to keep IPTables in place as a second line of defense, you should modify its rules to match the DigitalOcean Cloud Firewall settings. You can learn how to modify the IPTables rules by following the Iptables Essentials: Common Firewall Rules and Commands guide.

Viewing Running Services

If you are experiencing connection problems, it's useful to verify which services are actually running on your Droplets. Sometimes a service might go down or be under heavy load, which can cause some of the same symptoms as a closed port.

On your Droplets, you can view the services currently bound to a port by typing:

  • sudo netstat -plunt
Output
Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 13704/nginx -g daem tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1762/sshd tcp6 0 0 :::80 :::* LISTEN 13704/nginx -g daem tcp6 0 0 :::22 :::* LISTEN 1762/sshd

Check the port numbers and interfaces in the Local Address section against your expected services. If one of your expected services is not present in the netstat output, your service isn't running. This would cause connection attempts to fail.

Evaluating Connectivity From an External Source

Gathering information about the state of your DigitalOcean Cloud Firewall rules and any host-based firewall rules is important to understand the intended network access. However, it's also important to validate the actual traffic filtering behavior with tests.

By installing tools like nmap on an outside host, we can check which ports we can connect to on our Droplets. If you have Firewall rules that limit access based on the originating host, you may have to install nmap on multiple machines (those in the allowed group and those outside) in order to get a complete picture.

Note: If you only wish to allow connections from certain hosts, targeting Firewall rules with tags will make management and testing significantly simpler. You can install nmap on a single test Droplet and then add or remove tags from that server to change which rules match the connection request.

For more complete information about scanning hosts with nmap, follow our guide on How To Test your Firewall Configuration with Nmap and Tcpdump.

Installing Nmap

To start, you will need to install nmap on a Droplet that is not a part of the infrastructure you wish to scan.

To install nmap on an Ubuntu or Debian testing Droplet, type:

  • sudo apt-get update
  • sudo apt-get install nmap

To install nmap on a CentOS testing Droplet, type:

  • sudo yum install nmap

You should now be able to check the target infrastructure.

Scanning TCP Ports

Once the scanner is installed, you can scan for open TCP ports by typing:

  • sudo nmap -sS -Pn -p- -T4 -vv --reason Droplet_IP_address

The scan will attempt a SYN TCP connection to every port on the target Droplet. If you want to limit the number of scanned ports in order to speed up the scan, you can replace the -p- option with a range or sequence of ports instead:

  • sudo nmap -sS -Pn -p 1-5000 -T4 -vv --reason Droplet_IP_address

If you remove the -p option entirely, nmap will scan the 1000 most common ports, which is good when you have limited time or are only interested in the state of common service ports:

  • sudo nmap -sS -Pn -T4 -vv --reason Droplet_IP_address

If you have rules that limit traffic according to origin, make sure to run this test against your target infrastructure from a host where connections are supposed to be allowed and again from a host where connections are supposed to be denied.

Scanning UDP Ports

Because UDP is a connectionless protocol, scanning with UDP is significantly more error prone and time consuming than TCP scans. Since the Cloud Firewalls block traffic by silently dropping offending packets, it is difficult or impossible to distinguish between open and filtered UDP ports by port scanning.

This is beneficial in most situations as it makes it more difficult for outside parties to understand the services running on your infrastructure. However, this interferes with your ability to audit your UDP Firewall rules and so it cannot be used reliably for this purpose.

You can read more about the limitations of UDP scanning in our article on testing firewalls with nmap and tcpdump.

General Troubleshooting Information

So far, we've demonstrated how to gather information about your network policies and audit them. If you are still having trouble tracking down the source of your problems, some of the information in this section may help.

Connections Blocked by DigitalOcean Cloud Firewalls Will Timeout

The message you receive when attempting to connect to your server often contains valuable information about the source or type of problem you are experiencing. The exact user-facing message may be different depending on which type of client you are using. For example, SSH clients will have different failure messages than web browsers, and web browsers will each have their own style of messaging.

When DigitalOcean Cloud Firewalls filter traffic, they drop packets that do not match any exception rules. Eventually, after repeated failed attempts, the client connection typically times out. If you see a connection timeout message when attempting to connect to your Droplets, it may be caused by your DigitalOcean Cloud Firewalls rules.

Keep in mind that many other types of scenarios can also cause connection timeouts. For instance, many host-based firewalls also filter by dropping packets. This is also a status you may see if the service you are trying to reach is not operating or if the server itself has gone down. If a connection was redirected incorrectly by a routing component, this could also be the response.

If you see a timeout error, your Cloud Firewalls policy may be responsible. Double-check your rules to make sure that the appropriate inbound and outbound traffic is permitted.

If you immediately receive an error message that is not a timeout, the request was blocked by something other than Cloud firewalls. Some of the following messages fall into this category:

  • connection refused
  • destination unreachable
  • port unreachable
  • host unreachable
  • net unreachable
  • protocol unreachable
  • net prohibited
  • host prohibited
  • admin prohibited
  • no route
  • address unreachable
  • TCP reset

These are all indicators that the connection was denied by some other piece of software. For instance, most host-based firewalls include an option to deny traffic with an ICMP rejection message instead of silently dropping packets. Clients that receive these messages are able to error immediately instead of retrying until a timeout limit is reached.

DigitalOcean Cloud Firewalls do not send rejection messages, so if you see one, it means that your request is being blocked somewhere else.

DigitalOcean Cloud Firewall Rules are Additive

DigitalOcean Cloud Firewalls drop all traffic as a policy. To allow traffic to pass through, you must define exceptions by adding rules.

Because all rules are exceptions to the default drop policy, the ruleset is entirely additive. This means that once you allow traffic in one rule, it is impossible to lock it back down in another rule.

This can lead to unexpected results when more than one rule applies to a connection, especially if they come from separate Cloud Firewalls. If the connection matches any rule, it is able to pass through the firewall. You can view the aggregate rules applied to your Droplet by checking out the Networking section on the Droplet's detail page.

Restricting Connections by Source or Destination Requires Static IP Addresses

Cloud Firewalls are able to restrict inbound traffic based on the originating IP address and outbound traffic based on the destination IP address. This works well when the secondary party is another server with a static IP address.

However, you may run into trouble if you try to use this functionality to restrict connections to your home connection. Many internet service providers rotate customer IP addresses over time. This means that although a rule limiting connections to your home IP address may work now, it may stop working in the future.

Keep this in mind if you have Cloud Firewall rules that limit connections. If the address or network range is not stable, the rule will not function correctly over time.

Applications Sometimes Require Additional Network Access

It is important to keep in mind that some applications and services require broader network access than the regular listening port. Not accounting for these situations can lead to difficult to debug issues.

Some examples of additional required network access that might not be immediately obvious:

  • DigitalOcean Monitoring Agent: The DigitalOcean Monitoring agent requires outbound TCP access on ports 80 and 443 to authenticate to the Monitoring service and to send metrics. If you are restricting outbound traffic on these ports, DigitalOcean Monitoring will no longer work.
  • DNS Lookups: Many applications require DNS access to function correctly. Blocking outbound DNS (TCP and UDP port 53) may lead to many difficult to diagnose issues.
  • Service Coordination: Many applications use additional ports to coordinate with other servers or to accept special types of traffic. For instance, databases use designated ports for clustering or replication.

Not accounting for these additional requirements can cause problems, so be sure to check which ports your services are using with sudo netstat -plunt.

Conclusion

In this guide, we've covered how to find information about your DigitalOcean Cloud Firewall settings, how to check for the existence of conflicting rules on your Droplets, and how to audit connectivity from an external location. We've also discussed some important information to keep in mind when designing or troubleshooting your network connectivity.

While DigitalOcean Cloud Firewalls are designed to be simple to use, managing network access can be difficult. Learning how to diagnose problems can help you identify the cause of connection problems. To help avoid some common problems with firewalls, consider adopting some of the practices recommended in our How To Organize Your DigitalOcean Cloud Firewalls guide.

Creative Commons License