Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
Is it possible to configure an sftp user for uploading and editing but NOT downloading files? Question Question
Set umask for another user Question Question

Question

Why my droples's ssh key changed?

Posted April 12, 2017 8.6k views
SecurityUbuntu 16.04

Why my droples’s ssh key changed?
Changed in April 11, 18:49.

Screenshot:
https://goo.gl/photos/aGeqxtcS79Js4cRE7

Add a comment
neveralso57aff03a4fe5d19bf
By:
neveralso57aff03a4fe5d19bf

Related

Join the DigitalOcean Community Community

Join 1M+ other developers and:

  • Get help and share knowledge in Q&A
  • Subscribe to topics of interest
  • Get courses & tools that help you grow as a developer or small business owner
 Join Now
Is it possible to configure an sftp user for uploading and editing but NOT downloading files? Question Question
Set umask for another user Question Question
Add a comment
1 comment
  • porkchop April 13, 2017
    Reply Report

    Same here. However…

    Not only did my SSH host key change, but my system gained an entirely new “identity” to NewRelic as well… So the old identity appears to have gone away, and a “new” server exists in NewRelic. I’ve never seen that happen before on a server being monitored by NewRelic.

    You guys have pulled some weird stuff here.

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
Submit an Answer
8 answers
AliceLovesReptiles April 12, 2017

For anyone in the SFO2 region asking about the key update, there was an outage in the SFO2 region (April 11 2017) and the remote host identification error you are seeing is normal. The host key was updated due to cloud-init as a part of the outage.

You will want to remove the offending line from your ~/.ssh/known_hosts file so that the new key can be accepted.

Reply Report
hansen April 12, 2017

@neveralso57aff03a4fe5d19bf This is the community forum, there’s no “you” here.

@neveralso57aff03a4fe5d19bf @tlowrey Have you tried creating an support ticket to get an official answer? Please report back here with the answer.

Reply Report
  • tlowrey April 12, 2017

    Yes indeed, I have opened a support ticket, and they keep giving me the run-around with excuses and incompetency.

    They claim it’s totally normal, and our droplet must not have been restarted since it was created.

    Except that’s patently false. Not only has our droplet been restarted 100s of times since inception, it has also been rebooted and resized from the DO admin panel since they claim it was last rebooted. Never mind also that I have months of wtmp data showing that my server has been rebooted numerous times even in just the last few months.

    This still does not explain why our host keys were reset, and why they created a “debian” user on our server… after… a power outage. This is pure insanity.

    Reply Report
    • hansen April 12, 2017

      @tlowrey I totally agree, that’s not a good answer. Let’s see if we can get another one from Darian (the Platform Support Advocate)

      @dwilkin Hi Darian, do you have an answer about timestamps of keys being modified after SFO2 crash? I’m not sure if you’re the right one to ask, otherwise please let us know who might be the right one to answer this.

      Reply Report
      • dwilkin April 13, 2017

        Hey @hansen! Sorry I’m just getting back to you on this, happy to provide what clarity I can here. These keys were reset by cloud-init on reboot, though the reasons why may vary. Honestly cloud-init has some funky behavior and it’s not my strong suit. Here are a few things I can say:

        1) The Droplets which experience this weren’t rebooted at the Hypervisor level in a long time. While some may have gone through internal reboots, they don’t update libvirt and pick up new configurations from the Hypervisor unless they tell the Hypervisor that they’re shutting down. So @tlowrey that’s likely why your Droplet just now picked up the new configuration. You can see the prior Hypervisor-level power-off events in your Droplet’s history page:

        https://cloud.digitalocean.com/droplets/DROPLETID/history

        If it’s not listed there, the Hypervisor didn’t know about it.

        2) What information can be found on why these changes are occurring is located in /var/log/cloud* and /etc/cloud. You’ll likely find information on those host key changes in /var/log/cloud* log files, as well as why they were regenerated.

        3) The debian user bug is also cloud-init related. We initially dealt with it in October of last year regarding the way Debian snapshots interacted with cloud-init, causing the “default” user to be set to “debian” instead of root, causing cloud-init to attempt create that user on boot when it sees it doesn’t exist. We fixed it in late December for our systems, so I suspect this problem is only occurring on Debian Droplets created from snapshots before the new year that were rebooted during the SFO2 outage.

        I do encourage everyone who’s experiencing either of these issues to make a ticket so we can take a closer look at your logs specifically and determine if this is a new problem or just remnants from the old one. Please include logs and anything you think is relevant, the more information the better.

        Finally, I’d like to note that these issues would have happened the next time your Droplet rebooted at the hypervisor level regardless. It just seems like something happened all at once because we had an unprecedented number of Droplets reboot all at once. We have seen this host key issue before, it’s just that this is the first time it’s happened at scale due to the large number of reboots. I can assure you your data is just as secure as it was prior to the outage, and that’s our top priority.

        Please let us know if you have any additional questions, and never hesitate to reach out via support ticket.

        Regards,
        Darian
        Platform Support Advocate

        Reply Report
        • raychenon May 4, 2017

          These keys were reset by cloud-init on reboot…
          It just seems like something happened all at once because we had an unprecedented number of Droplets reboot all at once.

          As you admit a number of droplets rebooted without any user’s interventions.
          It happened to me 3 times in a month.
          Is there a way to prevent the reset of ssh keys by cloud-init ?

          After a reboot, the Droplet stack & apps are still there , so why SSH keys cannot be persisted too ? noob question

          Reply Report
          • dwilkin May 9, 2017

            Sorry for the confusion on this, I was referring to an outage in SFO2 that affected a large number of users, not regular power-off events for individual Droplets. If you’re having trouble with your Droplet powering off, please reach out via a ticket and we’d be happy to take a look!

            Regards,
            Darian
            Platform Support Advocate

            Reply Report
      • porkchop April 13, 2017

        Or my instance getting an entirely new identity to NewRelic all of a sudden?

        Reply Report
hansen April 12, 2017

Hi @neveralso57aff03a4fe5d19bf

Did you upgrade your system at that time - or do you have auto-update on, then it could modify the file date.

Otherwise, I would probably make a complete key change to ensure you are the only one accessing the system.

Reply Report
  • neveralso57aff03a4fe5d19bf April 12, 2017

    No, I couldn’t connect to my droplet yesterday because of the SFO2 connection issue. I didn’t upgrade my system and auto-update is always off.

    The file date is not the problem, the problem is that the file content has been changed! And I am not notified! Does this means that you can modify my droplet’s data without my authorization?

    Reply Report
    • tlowrey April 12, 2017

      The file date is not the problem, the problem is that the file content has been changed! And I am not notified! Does this means that you can modify my droplet’s data without my authorization?

      Yeah… this is what I’m upset about as well. Luckily ours was a QA development machine, and wasn’t used in production, but this is still very unsettling.

      Reply Report
      • dwilkin April 13, 2017

        I totally get that it’s concerning. As I mentioned elsewhere in this thread, these changes were made by cloud-init and not actively by an external process at DigitalOcean. We don’t mess with the data inside your Droplet after it’s created, only installed applications do that. Cloud-init is the application in question here, and you can find information on why in /var/log/cloud*

        Reply Report
        • tlowrey April 13, 2017

          We don’t mess with the data inside your Droplet after it’s created

          How do you reset passwords then?

          Reboots at the hypervisor level should be analogous to reboots from the “virtual machine” level. Data we have stored should not change from a hard reboot vs a soft reboot.

          Sounds to me like what really happened is y'all “migrated” our instances, which caused cloud-init to kick back off as if we were creating an instance from a snapshot. This is not the same as simply booting back up after a power loss, and I think this is why many users are so confused.

          Reply Report
          • dwilkin April 13, 2017

            How do you reset passwords then?

            Good question! This is more akin to you messing with the data inside your Droplet post-creation, we just provide a tool that allows it.

            Reboots at the hypervisor level should be analogous to reboots from the “virtual machine” level.

            Unfortunately this isn’t the case, if the reboot signal never reached the Hypervisor several actions never occur. To make sure you’re performing a Hypervisor-level reboot in the future, I’d encourage using:

            sudo shutdown -r now

            Sounds to me like what really happened is y'all “migrated” our instances

            No instances were migrated as a result of this issue, but please keep in mind that Droplets are virtualized. This means when the Hypervisor is powered off suddenly and comes back online, it’s to a whole new KVM instance. From your Droplet’s perspective, it went off and came back online with a totally new instance. As a result, it behaves in the same way it would after a migration, even though no migration occurred.

            Regards,
            Darian
            Platform Support Advocate

            Reply Report
    • dwilkin April 13, 2017

      I absolutely understand your concern here! But this isn’t something that was actively modified, it was a host key that cloud-init regenerated on boot for some reason or another. You can find more information on why in /var/log/cloud*.

      Long story short, this was an application within your Droplet that made those changes, nothing external reached in and adjusted anything. Were you to remove cloud-init from your Droplet (not really recommended) you wouldn’t see this happen again.

      Reply Report
tlowrey April 12, 2017

This happened to us as well. Our droplet rebooted twice about 9 hours ago, and now the host keys are changed. This is definitely NOT normal.

Reply Report
tlowrey April 12, 2017

How many of you are on Debian instances?

Reply Report
tlowrey April 12, 2017

Some of us on IRC are reporting same thing… SFO2, Debian, host keys reset. I have a colleague and another person on IRC who are on Ubuntu and their host keys did not reset.

Reply Report
tlowrey April 12, 2017

Our keys were changed Apr 11 10:44 (EDT) for what that’s worth…

Reply Report
tlowrey April 12, 2017

@AliceLovesReptiles Power outages do not cause SSH host keys to reset. It would be one thing if this was shared hosting, but theres are virtual machines, no?

Reply Report

Related

Looking for something else?

Ask a new question Search for more help