SSH through IPv6 times out

I first noticed the problem that whenever I would try to ssh into my droplet using the domain name, it would take about two minutes. When I used ssh -vvv, I found that it was the IPv6 causing the problem-- It would time out when trying the IPv6 address, then fall to the IPv4 and work fine–

debug1: Connecting to example.com [****:****:***:**::****:8001] port 22.
debug1: connect to address ****:****:***:**::****:**** port 22: Connection timed out
debug1: Connecting to example.com [***.***.**.240] port 22.
debug1: Connection established.

(I generalized the message. It’s not exactly a secret site, though.)

And when I try to connect using only the IPv6 address instead of the domain, it fails. (Works fine with IPv4 address.)

I went through the DO SSH troubleshooting article, but it didn’t make any difference. Where can I go to try to troubleshoot this?

(If it’s relevant, I’m using nginx on this server. I have another server that uses nginx, but it doesn’t have this problem.)