Question

DNS resolver inconsistency between Ubuntu 18.04.3 (LTS) x64 and 20.04 (LTS) x64 droplets

Posted June 4, 2020 3.9k views
DNSSystem ToolsUbuntu 20.04

Hi.

I believe there is a problem with the default configuration of the Ubuntu 20.04 LTS droplet on DigitalOcean…

As I’m sure you’re aware, back in Ubuntu 16.04, the package resolvconf managed the contents of /etc/resolv.conf (hence the DO NOT EDIT warnings of the past we all remember)

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

Since Ubuntu 18.04 however, the systemd-resolved package has replace resolvconf.

The systemd-resolved package runs a local stub DNS resolver on 127.0.0.53. The intention is that /etc/resolv.conf points to 127.0.0.53 and so any program on the system attempting to perform name resolution will hit the local DNS stub resolver, at which point systemd-resolved will intercept the query and forward or answer the question as configured, and as appropriate.

This means the user can specific different DNS server search orders, prior to, for example using the static, or DHCP assigned DNS servers, or express a preference to use a DNS server available via a VPN tunnel for example.

In the default Ubuntu 18.04.3 (LTS) x64 droplet, this is configured as expected. /etc/resolv.conf contains the directive which directs queries to the local stub resolver:

nameserver 127.0.0.53
options edns0

And in the static IP address configuration defined in /etc/netplan/50-cloud-init.yaml we see the DigitalOcean nameservers statically assigned to the eth0 interface:

nameservers:
   addresses:
   - 67.207.67.3
   - 67.207.67.2

The Ubuntu 18.04 DigitalOcean droplet works as expected.

The problem is that the Ubuntu 20.04 LTS DigitalOcean droplet does not follow this pattern.

If we create a new Ubuntu 20.04 LTS droplet, we see that, as before /etc/netplan/50-cloud-init.yaml contains the static nameserver addresses

nameservers:
   addresses:
   - 67.207.67.3
   - 67.207.67.2

But /etc/resolv.conf is not using the local stub resolver and instead has been configured to use Google name servers.

nameserver 8.8.8.8
nameserver 2001:4860:4860::8888
nameserver 2001:4860:4860::8844

If we then interrogate the system using resolvectl dns we can see the inconsistency come to light.

Global: 8.8.8.8 2001:4860:4860::8888 2001:4860:4860::8844
Link 2 (eth0): 67.207.67.3 67.207.67.2

On the one hand, eth0 interface is statically assigned to use the DigitalOcean name servers, and on the other, there is a global configuration override to ignore systemd-resolved and the local stub resolver and instead direct all DNS queries out to Google’s name servers.

This configuration seems sub-optimal for a number of reasons.

  1. The configuration only points to one of Google’s IPv4 name servers.
  2. Any transit or connectivity problems between DigitalOcean and Google knocks out name resolution for DigitalOcean customers across the board running Ubuntu 20.04 LTS and is then out of your control as a company and engineering team.
  3. Lastly, but by far most importantly, this configuration completely overrides any configuration made to systemd-resolved as applications on the the local system fail to interact with the local stub resolver.

May I request that you investigate why the default behaviour of the DNS resolver has changed in the DigitalOcean Ubuntu 20.04 LTS droplet and either

a) Fix the default configuration of /etc/resolv.conf to match 18.04 LTS or,
b) Help me (and the community) understand your reasoning for this assumption-breaking change.

Many thanks in advance.

Kind regards,
Marc

4 comments
  • I’m surprised this hasn’t been answered yet. Have you escalated this to the support team at all yet?

  • Hi Jon, thanks for following up. I’ve not escalated (in truth I’m not sure how)

  • We also run into this problem when investigating intermediate host resolution errors when running migrations via DigitalOcean. We may migration thousands of objects per second, and with this setup it results in thousands of DNS requests to Google server to resolve same name, which does not seem to like it after a while.

  • Hey everyone,

    My name is Ethan, I work on the Cloud Operations team here at DO. This issue was escalated internally to our Engineering team who will take a look at why we have Ubuntu 20.04 configured in this manner. As I see updates internally, I’ll do my best to ensure that new information boils up here as well.

    Kind Regards,
    Ethan F - Cloud Operations

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer

Hello!

My name is Daniel and I work on the team that manages how images are brought onto the DO platform for use. Our team has been working on improving how images are built and tested before they are released to everyone at DO. It looks like as part of that effort, we accidentally included an out of date etc/resolve.conf file. We have removed the file, and will ensure that this doesn’t happen again.

Sorry for the confusion,
Daniel.

Submit an Answer