We have a CoreOS dropplet and have manually setup ssh keys for it via the ~/.ssh/authorized_keys file. Once in a while (it happens about 2 times per month), the keys in that file would revert to the globally set ones via the DigitalOcean website inside the security setting page. Does anyone have any idea why this keeps happening?
The only thing I can think of is cloud-init somehow being executed again and in some very non-correct way running the installation script again and replacing the keys with the one in the DigitalOcean control panel connected to that droplet.
But I've never used the CoreOS.
Let's check if one of the DigitalOcean staff can help on this...
@jtittle is this something you know something about or would that be dwilkin or asb?
Is there a way to trigger the cloud-init manually? Maybe we are doing something wrong ourselves, but it seems unlikely.
When it resets the key, have you restarted the droplet just before or any other power/reboot commands?
Are you running cloud-init anywhere - it's a command (at least on Ubuntu) ?
Or are you restoring from backup/snapshot ?
Or are you doing any special API calls - either directly or via tools like doctl ?
If it's a no to everything, then I'm not sure what's going on and you have to wait for the DO techs to give feedback. Might not get an answer until Monday.
I know DO has a love/hate relationship with cloud-init. It's one of the reasons why we can take a snapshot and spin up the server somewhere else and it automatically gets the new IP - and also the reason for resizing the disk automatically - among other features.
But it was also one of the headaches when the SFO data center had a power outage (a couple of months ago) and suddenly keys were being reset and other havoc, when the systems were coming back online.
I don't think we explicitly call cloud-init anywhere. At least I didn't know about its existence until now, and I doubt my colleagues are using it either, since I am the one dealing with the cloud environments most of the time, but I'll ask. Do you know if cloud-init takes the ssh keys from DO's settings every time it is invoked, or does it use the ones that were active when the dropplet was initially created?
Also we have no snapshots or backups, this is more of a demo environment that we don't care too much about.
Additionally, it seems like when the ssh keys get reset, our containers get killed too, which suggests that it happens after a system crash of some sort, but the whole phenomenon doesn't occur often enough to be sure about all the details.
Sounds like a crash of some sort. I'm not sure if the command uptime exists on CoreOS - otherwise the dmsg file is reset on reboot and you can get the reboot time from that ls -l /var/log/dmesg.
Also have a look in https://cloud.digitalocean.com/settings/security (scroll down to History) to see if you can see the droplet being rebooted or any other activity, which usually involves the cloud-init process in some way.
Otherwise, let's see what the good techs have to say. I can understand it's an annoying problem, but I actually find this interesting, so I'll continue to follow this thread to see what the resolution might be.
My guess is it's actually an update. CoreOS Container Linux is designed to auto-update. It would fit the timeline described.
You can see what cloud-init is doing by following the instructions for using journalctl here: https://coreos.com/os/docs/latest/cloud-config.html
@pgburt But that would still mean that cloud-init is not configured correctly, since it should never-ever run the installation again on reboot/update.
Marking the authorized_keys file as immutable seems to prevent changes to it on reboot.
We did some more testing and it seems that every time we reboot the machine, it overwrites ~/.ssh/authorized_keys with the ssh keys that were active when the machine was initially created.
Marking the ~/.ssh/authorized_keys file as immutable seems to solve the issue.