How To Recover from File System Corruption Using Fsck and a Recovery ISO
Despite your best efforts, there are certain instances where your server might suffer file corruption and need recovery. Sometimes this happens when your VPS is powered off abruptly, or when a piece of software or hardware malfunctions suddenly.
Either way, there are steps you can take to try to recover your VPS, or at least your important files. We will go through the steps we recommend you taking in this guide.
Important Considerations and Risks
In any circumstance, good backups are the best way to prevent data loss. Implementing a reliable offsite backup solution and thoroughly testing it on a regular basis is the only way to guarantee the integrity of your important data.
While recovery options like
fsck are often useful, there is no guarantee that it will work correctly and success is often simply a matter of luck. The
fsck operation can occasionally cause data corruption on active disks. For this reason, the
fsck procedure outlined below operates on an unmounted file system to minimize this risk. Problems can still occur in cases of severe damage though, so consider these last-resort methods for data recovery.
Attempt Recovery with Fsck Kernel
Newer distributions including FreeBSD, CoreOS, Debian 8 and Ubuntu 15.04 cannot use the recovery kernel. If you are using one of these releases please proceed to the "Attempt Recovery with a Recovery ISO" section below.
The first step in trying to recovery your system from file corruption is to mount our recovery kernel that will allow you to run fsck, which is a file system checking utility. This can help find and fix errors in your filesystem.
Run Fsck on your Droplet
To begin, power off your Droplet in the safest way available to you. If you have access to the VPS through the command line, run this command:
If you are unable to log in and power the Droplet down gracefully, you can use the "Power Off" button in the control panel:
Once your Droplet is off, we will change the kernel by going to "Settings" and clicking on the "Recovery" tab. When you are ready, click on the "Mount Recovery Kernel" button:
After the kernel has been changed, you can boot the Droplet by clicking on the "Boot" button under the "Power" tab:
After the Droplet is started, you can click on the "Console Access" button in the console to access the Droplet. You won't be able to SSH into the VPS at this time because the server is booted with a minimal recovery kernel:
A terminal session will open in the current window and you will be logged into a minimal Linux environment.
From here, you should run the
fsck program to find and fix errors on the filesystem.
The method in which you invoke this command will depend on whether the Droplet was created with the "Enable VirtIO" box checked. If it was selected, your hard drive will be
/dev/vda1 depending on your system. You can find out which one your system is configured to use by typing:
If VirtIO was not selected, the hard drive will be
/dev/sda. By default, this box is selected.
If VirtIO was selected when you created the Droplet, run:
fsck -yf /dev/vda
fsck -yf /dev/vda1
If VirtIO was not selected, run:
fsck -yf /dev/sda
This will run the
fsck utility and attempt to fix any errors. After this is complete, you can power off the Droplet by typing:
Back in the control panel, click on "Settings" and select the "Kernel" tab again. From the drop-down menu, select the kernel that you were previously using:
Click the "Change" button again.
Check the Results
When the kernel change has completed, boot the Droplet using the same button in the power menu that you used before.
Access the Droplet from the "Console Access" button again.
If the Droplet boots up and it was unable to boot previously, then this is a good sign. If you ran into specific problems earlier that alerted you to the possibility of corruption, you should try these operations again to see if they are successful.
An important thing to do is check the
/lost+found directory. This is where fsck puts partially recovered files.
Sometimes, fsck is able to recover file data, but it cannot find a reference to the file on the filesystem. Basically, it is a file without a name. If fsck is faced with this situation, it places these files in the
/lost+found directory so that you can manually try to figure out what the file is.
After running fsck, look to see if anything has been placed in that directory:
cd /lost+found ls
If there are files in this directory, you will want to see if you can identify them. Often, these are files that you had deleted anyways, but were still being used when the system crashed. It is worth checking them though to be sure.
If your filesystem is noticeably broken still, or if the boot fails when changing back to the normal kernel, you will need to continue to the next section to continue recovery operations.
Attempt Recovery with a Recovery ISO
When recovery with the fsck kernel fails, you will need to go a step farther in your recovery efforts. The next step is to attempt to rescue the system with a recovery ISO.
While users do not have the ability to mount a recovery ISO from the console, the DigitalOcean support staff can mount one for your Droplet if you are experiencing problems and have attempted recovery with the kernel first.
Power off your Droplet in the safest way available to you. Use this command if you have access to the command line:
If you do not have access to the command line, just use the "Power Off" button from the control panel.
In the control panel, click on the "Support" button on the left-hand navigation bar. In the upper-right corner, click "New Ticket".
Fill out the message with information about what you have tried and that you would like to attempt recovery with the recovery ISO image:
When you receive a response that the recovery ISO has been enabled on your Droplet, boot the Droplet if it is still off and access your VPS by clicking on the "Console Access" button again.
Once connected to the console you should see a main menu like the one below:
Set Up Networking in the Recovery Environment
The standard ISO that DigitalOcean uses for recovery will configure networking automatically on droplets in regions which support droplet meta-data. In other regions you will have to enter your network details manually.
We will use the information under the "Public Network" section displayed beneath the console window during this configuration:
From the menu select
2 and press
Enter. If your droplet is in a region that does not support meta-data you will now be prompted to enter your
Gateway. The recovery environment will then configure your network interface with these details and test to ensure your Droplet is connected to the Internet.
Run fsck scan from the Recovery ISO
To run an fsck filesystem check and repair you can select
3 from the menu and press
Enter. The recovery environment will detect your disk image and attempt to run an fsck on it. It will report any errors and bring any issues to your attention.
Mount the Filesystem and Begin Recovery Measures
This Linux environment is running from the ISO image and not from your Droplet, so you will have to mount the filesystem into our environment in order to access your files. To do so, select
1 at the menu and press
Enter. Your disk image will be detected and mounted under
/mnt in the recovery environment.
Next, you will need to enter an interactive bash shell environment by selecting
7 from the menu and pressing
If you previously ran a filesystem check either from the recovery kernel or the recovery ISO, you can now check for any partially recovered files in the
Go to the
/mnt directory and you should see your Droplet's filesystem:
cd /mnt ls
bin/ etc/ lib/ media/ proc/ sbin/ sys/ var/ boot/ home/ lib64/ mnt/ root/ selinux/ tmp/ vmlinuz@ dev/ initrd.img@ lost+found/ opt/ run/ srv/ usr/
From here, we can check the
lost+found directory for partially recovered files:
cd lost+found ls
If this directory has files that were recovered by
fsck, you can attempt to move them back into place and restore them on the system. This may help your system boot and function correctly if these were important files.
If the files in
lost+found are unrecoverable, or if you do not want to try to restore the Droplet and only want to save some data, you can try to offload your files onto a remote machine (another Droplet, or another physical machine).
Using SFTP to Transfer Files off your Droplet
If you need to transfer files from your Droplet to your local computer you can do so from the recovery environment. If you are in the interactive shell you can type
exit to return to the main menu.
First, ensure that the main menu reports your filesystem is mounted and select
4 from the menu and press
Enter to enable an SSH server.
You will be prompted to create a temporary root password in order to access your Droplet. Note: This will not change the root password inside your Droplet.
Enter the new temporary password twice. After this, the recovery environment will install and configure an SSH server.
Now that the SSH service has been enabled it can be reached using an SSH or SFTP client. Using the Filezilla SFTP client, you would create a new connection with the following details substituting your Droplet's IP address and the temporary password you created with the values shown:
Host: your_droplets_IP Port: 22 Protocol: SFTP - SSH File Transfer Protocol Login Type: Normal User: root Password: TEMPORARY_PASSWORD
When you connect, you will start out in the directory
/root. Your Droplet's filesystem will be located in
/mnt. You can navigate to this directory to begin selecting files and transferring them to your local machine.
More information about using Filezilla can be found here.
Where Do You Go From Here?
Your options at this point depend on how successful your recovery operations were.
If your recovery went well and the Droplet itself is salvageable, you can power off from the command line again:
After it is done, you should message a support team member to remove the ISO image from the Droplet. You can then boot normally and continue to use the Droplet as you were before.
If you have transferred the important files to a new location and you are unable or unwilling to recover the original Droplet, you can simply destroy the instance from the control panel.
You can then create a new image and transfer the recovered data.
While filesystem corruption is never a good thing, it doesn't necessarily mean that all of your important data was lost. The success of your recovery operations comes down to a number of factors, such as how quickly the filesystem noticed the corruption, how widespread the issue was, and what files were affected.
When it comes down to it, simple recovery using automated tools is largely a function of luck. That being said, in many cases, fsck recovery is successful and you can go back to using your server without too much of a headache. Remember, keeping good backups is the most important step you can take to prevent data loss.