Why my droples’s ssh key changed?
Changed in April 11, 18:49.
Screenshot:
https://goo.gl/photos/aGeqxtcS79Js4cRE7
These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.
×For anyone in the SFO2 region asking about the key update, there was an outage in the SFO2 region (April 11 2017) and the remote host identification error you are seeing is normal. The host key was updated due to cloud-init as a part of the outage.
You will want to remove the offending line from your ~/.ssh/known_hosts file so that the new key can be accepted.
@neveralso57aff03a4fe5d19bf This is the community forum, there’s no “you” here.
@neveralso57aff03a4fe5d19bf @tlowrey Have you tried creating an support ticket to get an official answer? Please report back here with the answer.
Yes indeed, I have opened a support ticket, and they keep giving me the run-around with excuses and incompetency.
They claim it’s totally normal, and our droplet must not have been restarted since it was created.
Except that’s patently false. Not only has our droplet been restarted 100s of times since inception, it has also been rebooted and resized from the DO admin panel since they claim it was last rebooted. Never mind also that I have months of wtmp data showing that my server has been rebooted numerous times even in just the last few months.
This still does not explain why our host keys were reset, and why they created a “debian” user on our server… after… a power outage. This is pure insanity.
@tlowrey I totally agree, that’s not a good answer. Let’s see if we can get another one from Darian (the Platform Support Advocate)
@dwilkin Hi Darian, do you have an answer about timestamps of keys being modified after SFO2 crash? I’m not sure if you’re the right one to ask, otherwise please let us know who might be the right one to answer this.
Hey @hansen! Sorry I’m just getting back to you on this, happy to provide what clarity I can here. These keys were reset by cloud-init on reboot, though the reasons why may vary. Honestly cloud-init has some funky behavior and it’s not my strong suit. Here are a few things I can say:
1) The Droplets which experience this weren’t rebooted at the Hypervisor level in a long time. While some may have gone through internal reboots, they don’t update libvirt and pick up new configurations from the Hypervisor unless they tell the Hypervisor that they’re shutting down. So @tlowrey that’s likely why your Droplet just now picked up the new configuration. You can see the prior Hypervisor-level power-off events in your Droplet’s history page:
https://cloud.digitalocean.com/droplets/DROPLETID/history
If it’s not listed there, the Hypervisor didn’t know about it.
2) What information can be found on why these changes are occurring is located in /var/log/cloud* and /etc/cloud. You’ll likely find information on those host key changes in /var/log/cloud* log files, as well as why they were regenerated.
3) The debian user bug is also cloud-init related. We initially dealt with it in October of last year regarding the way Debian snapshots interacted with cloud-init, causing the “default” user to be set to “debian” instead of root, causing cloud-init to attempt create that user on boot when it sees it doesn’t exist. We fixed it in late December for our systems, so I suspect this problem is only occurring on Debian Droplets created from snapshots before the new year that were rebooted during the SFO2 outage.
I do encourage everyone who’s experiencing either of these issues to make a ticket so we can take a closer look at your logs specifically and determine if this is a new problem or just remnants from the old one. Please include logs and anything you think is relevant, the more information the better.
Finally, I’d like to note that these issues would have happened the next time your Droplet rebooted at the hypervisor level regardless. It just seems like something happened all at once because we had an unprecedented number of Droplets reboot all at once. We have seen this host key issue before, it’s just that this is the first time it’s happened at scale due to the large number of reboots. I can assure you your data is just as secure as it was prior to the outage, and that’s our top priority.
Please let us know if you have any additional questions, and never hesitate to reach out via support ticket.
Regards,
Darian
Platform Support Advocate
These keys were reset by cloud-init on reboot…
It just seems like something happened all at once because we had an unprecedented number of Droplets reboot all at once.
As you admit a number of droplets rebooted without any user’s interventions.
It happened to me 3 times in a month.
Is there a way to prevent the reset of ssh keys by cloud-init ?
After a reboot, the Droplet stack & apps are still there , so why SSH keys cannot be persisted too ? noob question
Sorry for the confusion on this, I was referring to an outage in SFO2 that affected a large number of users, not regular power-off events for individual Droplets. If you’re having trouble with your Droplet powering off, please reach out via a ticket and we’d be happy to take a look!
Regards,
Darian
Platform Support Advocate
Or my instance getting an entirely new identity to NewRelic all of a sudden?
@porkchop I’m not as familiar with how NewRelic designates instances, is it possible they use host keys? If so, that would explain how these issues are related.
Hi @neveralso57aff03a4fe5d19bf
Did you upgrade your system at that time - or do you have auto-update on, then it could modify the file date.
Otherwise, I would probably make a complete key change to ensure you are the only one accessing the system.
No, I couldn’t connect to my droplet yesterday because of the SFO2 connection issue. I didn’t upgrade my system and auto-update is always off.
The file date is not the problem, the problem is that the file content has been changed! And I am not notified! Does this means that you can modify my droplet’s data without my authorization?
The file date is not the problem, the problem is that the file content has been changed! And I am not notified! Does this means that you can modify my droplet’s data without my authorization?
Yeah… this is what I’m upset about as well. Luckily ours was a QA development machine, and wasn’t used in production, but this is still very unsettling.
I totally get that it’s concerning. As I mentioned elsewhere in this thread, these changes were made by cloud-init and not actively by an external process at DigitalOcean. We don’t mess with the data inside your Droplet after it’s created, only installed applications do that. Cloud-init is the application in question here, and you can find information on why in /var/log/cloud*
We don’t mess with the data inside your Droplet after it’s created
How do you reset passwords then?
Reboots at the hypervisor level should be analogous to reboots from the “virtual machine” level. Data we have stored should not change from a hard reboot vs a soft reboot.
Sounds to me like what really happened is y'all “migrated” our instances, which caused cloud-init to kick back off as if we were creating an instance from a snapshot. This is not the same as simply booting back up after a power loss, and I think this is why many users are so confused.
How do you reset passwords then?
Good question! This is more akin to you messing with the data inside your Droplet post-creation, we just provide a tool that allows it.
Reboots at the hypervisor level should be analogous to reboots from the “virtual machine” level.
Unfortunately this isn’t the case, if the reboot signal never reached the Hypervisor several actions never occur. To make sure you’re performing a Hypervisor-level reboot in the future, I’d encourage using:
Question
sudo shutdown -r now
Sounds to me like what really happened is y'all “migrated” our instances
No instances were migrated as a result of this issue, but please keep in mind that Droplets are virtualized. This means when the Hypervisor is powered off suddenly and comes back online, it’s to a whole new KVM instance. From your Droplet’s perspective, it went off and came back online with a totally new instance. As a result, it behaves in the same way it would after a migration, even though no migration occurred.
Regards,
Darian
Platform Support Advocate
I absolutely understand your concern here! But this isn’t something that was actively modified, it was a host key that cloud-init regenerated on boot for some reason or another. You can find more information on why in /var/log/cloud*.
Long story short, this was an application within your Droplet that made those changes, nothing external reached in and adjusted anything. Were you to remove cloud-init from your Droplet (not really recommended) you wouldn’t see this happen again.
@AliceLovesReptiles Power outages do not cause SSH host keys to reset. It would be one thing if this was shared hosting, but theres are virtual machines, no?
Same here. However…
Not only did my SSH host key change, but my system gained an entirely new “identity” to NewRelic as well… So the old identity appears to have gone away, and a “new” server exists in NewRelic. I’ve never seen that happen before on a server being monitored by NewRelic.
You guys have pulled some weird stuff here.