How to Recover from File System Corruption Using Fsck and a Recovery ISO

File system corruption on your server can happen when software or hardware malfunctions or when your server is abruptly powered off. The success of your recovery operations comes down to a number of factors, such as how quickly the filesystem noticed the corruption, how widespread the issue was, and what files were affected.

The best way to prevent data loss in these situations is with good backups. Implementing a reliable offsite backup solution and thoroughly testing it on a regular basis is the only way to guarantee the integrity of your data.

However, you can also use DigitalOcean’s recovery ISO to run fsck, a filesystem check program. This recovery option can be useful, but it comes with risks and isn’t guaranteed to work, so consider it a last resort method of data recovery.

Step 1 — Boot into the Recovery ISO

First, power off your Droplet in the safest way available to you. If you have access to the command line, you can use the poweroff command:

sudo poweroff

If you do not have access to the command line, you can power off the Droplet using the toggle on the Droplet’s page in your control panel.

In the control panel, click Recovery on the left, then click the Boot from Recovery ISO box:

Next, toggle the power button to power your Droplet back on. When it’s on, access your Droplet by clicking on Access to the left, then clicking the Launch Console button. You’ll see a menu like this:

DigitalOcean Recovery Environment v 1.5

This image has been mounted by the DigitalOcean Support Team
When you have completed your work in the recovery environment
update your support ticket to request that your Droplet be booted
to it's disk

----------------------------------------


1. Mount your Disk Image [Not Mounted]
2. Configure Networking [Not Configured]
3. Check Filesystem
4. Enable SSH Server [Not Enabled]
5. Reset Droplet Root Password
6. Configure Keyboard
7. Interactive Shell [/bin/bash]

Choose (1-7) and press Enter to continue.

The recovery ISO provides a number of options.

Step 2 — Run fsck and Check for Files

fsck can occasionally cause data corruption on active disks, so only attempt this on an unmounted file system. To run fsck, select 3 from the menu and press ENTER. The recovery environment will detect your disk image and attempt to run an fsck on it. It will report any errors and bring any issues to your attention.

This Linux environment is running from the ISO image, not from your Droplet, so you will have to mount the filesystem into your environment in order to access your files. To do so, select 1 at the recovery menu and press ENTER. Your disk image will be detected and mounted under /mnt in the recovery environment.

Next, you will need to enter an interactive bash shell environment by selecting 6 from the menu and pressing ENTER. If you previously ran a filesystem check from the recovery ISO, you can now check for any partially recovered files in the /mnt/lost+found directory.

Move to the /mnt directory.

cd /mnt
ls

You should see your Droplet’s filesystem:

bin/    etc/        lib/        media/  proc/   sbin/       sys/    var/
boot/   home/       lib64/      mnt/    root/   selinux/    tmp/    vmlinuz@
dev/    initrd.img@ lost+found/ opt/    run/    srv/        usr/

From here, you can check the lost+found directory for partially recovered files:

cd lost+found
ls

Your options at this point depend on how successful your recovery operations were.

Step 3 — Finish Recovery

If the lost+found directory has files that were recovered by fsck, you can attempt to move them back into place on the system. This may help your system boot and function correctly if they were important files. If the recovery went well and the Droplet itself is salvageable, you can power off from the command line again:

poweroff

Then, back in the control panel, click Recovery and then choose the radio button for Boot from Hard Drive. When you power the Droplet back on, it will attempt to boot from its own disk.

If the files in lost+found are unrecoverable, or if you do not want to try to restore the Droplet and only want to save some data, you can offload your files onto a remote machine (like another Droplet or another physical machine). You can use SFTP from the command line or with an SFTP client like Filezilla. Make sure to use the temporary password in the recovery menu when you set up the connection.

You can also attempt to chroot into the filesystem if you wish to try more advanced recovery steps. For example, this may allow you to start MySQL so you can attempt to export your databases. However, this may not always work, depending upon the state of the filesystem.

Conclusion

When it comes down to it, simple recovery using automated tools is largely a function of luck. fsck recovery can help get your server back or at least get your files, but keeping good backups is the most important step you can take to prevent data loss.