Question

CoreOS manually set SSH keys keep resetting.

Posted June 25, 2017 3.3k views
Configuration ManagementCoreOS

We have a CoreOS dropplet and have manually setup ssh keys for it via the ~/.ssh/authorized_keys file. Once in a while (it happens about 2 times per month), the keys in that file would revert to the globally set ones via the DigitalOcean website inside the security setting page. Does anyone have any idea why this keeps happening?

Add a comment
emilenikolov
By:
emilenikolov
Add a comment
2 comments

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
1 answer
hansen June 25, 2017

Hi @emilenikolov

The only thing I can think of is cloud-init somehow being executed again and in some very non-correct way running the installation script again and replacing the keys with the one in the DigitalOcean control panel connected to that droplet.
But I’ve never used the CoreOS.

Let’s check if one of the DigitalOcean staff can help on this…
@jtittle is this something you know something about or would that be dwilkin or asb?

Reply Report
  • emilenikolov June 25, 2017

    Is there a way to trigger the cloud-init manually? Maybe we are doing something wrong ourselves, but it seems unlikely.

    Reply Report
    • hansen June 25, 2017

      @emilenikolov

      When it resets the key, have you restarted the droplet just before or any other power/reboot commands?
      Are you running cloud-init anywhere - it’s a command (at least on Ubuntu) ?
      Or are you restoring from backup/snapshot ?
      Or are you doing any special API calls - either directly or via tools like doctl ?

      If it’s a no to everything, then I’m not sure what’s going on and you have to wait for the DO techs to give feedback. Might not get an answer until Monday.

      I know DO has a love/hate relationship with cloud-init. It’s one of the reasons why we can take a snapshot and spin up the server somewhere else and it automatically gets the new IP - and also the reason for resizing the disk automatically - among other features.
      But it was also one of the headaches when the SFO data center had a power outage (a couple of months ago) and suddenly keys were being reset and other havoc, when the systems were coming back online.

      Reply Report
      • emilenikolov June 25, 2017

        I don’t think we explicitly call cloud-init anywhere. At least I didn’t know about its existence until now, and I doubt my colleagues are using it either, since I am the one dealing with the cloud environments most of the time, but I’ll ask. Do you know if cloud-init takes the ssh keys from DO’s settings every time it is invoked, or does it use the ones that were active when the dropplet was initially created?

        Also we have no snapshots or backups, this is more of a demo environment that we don’t care too much about.

        Additionally, it seems like when the ssh keys get reset, our containers get killed too, which suggests that it happens after a system crash of some sort, but the whole phenomenon doesn’t occur often enough to be sure about all the details.

        Reply Report
        • hansen June 25, 2017

          @emilenikolov

          Sounds like a crash of some sort. I’m not sure if the command uptime exists on CoreOS - otherwise the dmsg file is reset on reboot and you can get the reboot time from that ls -l /var/log/dmesg.

          Also have a look in https://cloud.digitalocean.com/settings/security (scroll down to History) to see if you can see the droplet being rebooted or any other activity, which usually involves the cloud-init process in some way.

          Otherwise, let’s see what the good techs have to say. I can understand it’s an annoying problem, but I actually find this interesting, so I’ll continue to follow this thread to see what the resolution might be.

          Reply Report
Submit an Answer

Related

Looking for something else?

Ask a new question Search for more help