Fedora 33 -- how to persist DNS settings via /etc/resolv.conf?

I contacted the DO support. This is a known issue and they are working on it. With the default settings in Fedora 33, they suggest this temporary workaround:

sudo resolvectl dns eth0 8.8.8.8 4.4.4.4

This command must be run after each reboot. I put the following lines in the droplet’s user data:

#cloud-config
bootcmd:
- [ resolvectl, dns, eth0, "8.8.8.8", "4.4.4.4" ]

This solved the issue for me. You can check the status with resolvectl. See also mcatanzaro’s answer above :-)

Kudos to the DO team for the quick response!

TL;DR: The simplest solution is to force use of your preferred DNS servers in /etc/systemd/resolved.conf. For example:

DNS=67.207.67.2 67.207.67.3

Then restart systemd-resolved:

$ sudo systemctl restart systemd-resolved.service

Now you’re good to go! Ensure it’s working withresolvectl query example.com (or with dig).

Now, for the longer answer: I’m afraid looking at RHEL 8 documentation is going to mislead you here. In RHEL 8 and Fedora 32, /etc/resolv.conf was managed by NetworkManager. But in Fedora 33, it is now managed by systemd-resolved, and is mostly ignored by default: software that uses glibc for name resolution – which is most software – will not read /etc/resolv.conf at all. More info here.

DigitalOcean’s freshly installed Fedora 33 droplets default to using Cloudflare’s 1.1.1.1 as fallback DNS because no other DNS servers are configured. Problem is, this is not really ever supposed to happen in practice: the assumption is that users would almost always receive a more specific DNS server via DHCP that would override the fallback DNS servers. But it seems that does not happen with DigitalOcean’s droplets. That is arguably a misconfiguration to be addressed – for instance, I notice that DigitalOcean’s Ubuntu droplets default to using DigitalOcean’s 67.207.67.2 and 67.207.67.3 DNS servers instead, which is better – but it’s not initially a big deal because Cloudflare is fine. Problem comes after your first system update. Next time you reboot (or restart systemd-resolved) after your first update, you’ll have no more DNS servers configured at all. Why? Because some users complained about fallback to Cloudflare and Google, so the fallback was removed in systemd-246.7-1.fc33. Consequence of that is that anyone relying on fallback DNS – including DigitalOcean’s F33 droplets – now no longer has any DNS configured at all after you update systemd. There are several possible ways to fix this, but since cloud servers don’t need any fancy dynamic configuration like desktops do, I suggest simply editing /etc/systemd/resolved.conf as shown above.

My opinion is that this problem is a bug on DigitalOcean’s end. Although I’m not a fan of Fedora removing the fallback DNS servers – and in retrospect, changing the behavior post-release was a clear mistake, oops! – it’s surely not intended for the fallback servers to be used in the first place. I see that DigitalOcean’s Fedora 32 droplets create a hardcoded /etc/resolv.conf using cloud-init:

; Created by cloud-init on instance boot automatically, do not edit.
;
nameserver 67.207.67.2
nameserver 67.207.67.3

Clearly that doesn’t work anymore. Ideally DigitalOcean would figure out how its Ubuntu droplets get configured with DigitalOcean’s DNS servers, then try to replicate that with Fedora’s droplets, since Ubuntu and Fedora 33 both use systemd-resolved and both use NetworkManager to configure systemd-resolved, so configuration should be almost identical (with the exception that Ubuntu still reads /etc/resolv.conf, but that shouldn’t matter for this).

PS. Just to add, this has nothing to do with my firewall.