The single most common reason we see users lose their data forever is a RAID rebuild failure. Users are usually inclined to simply remove the bad disk and rebuild again. This is essentially the same as destroying the RAID. Here's what to do next:
If you are rebuilding to a faulty disk or building in the wrong order, you could be overwriting your data permanently. Immediately stop and shut down the system. Don't change anything until you have had this evaluated.
There is no shortage of RAID emergencies where a rebuild stalls or was stopped (or even worse) leaves the array unable to mount. Knowing how RAID 5 works may help understand why rebuilding an array after it fails is so dangerous. Your RAID stores each file piece across multiple drives, but also includes a rotating block of parity data for each stripe. When a RAID loses a drive, parity lets it rebuild the missing data from the surviving drives, but no redundant information is left. In other words, once one drive is dead, the array has no safety net. It is now degraded and runs with no redundancy. The rebuild process is by far the most stressful operation a RAID can perform. Thus, when a RAID is running with no safety nets during a rebuild, a second problem will likely appear.
Nearly all rebuild failures boil down to one of these issues and they are all caused by the fact that the RAID is working flat-out with no fault tolerance remaining.
The rebuild will look at every sector of every available (surviving) disk. Because those disks have been used as long and have worn out equally as the dead disk, the constant usage puts enough strain on a marginal disk causing it to fail, and now two disks are down.
In order to build the stripe again, the controller has to read all the remaining data. If the controller hits an unreadable sector without any redundant sectors left to help fill in missing sectors, it cannot reconstruct the stripe; therefore, it will abort the rebuild.
The controller may kick a consumer (healthy) disk out of the array because it took too much time reading through a sector that was slower than normal; thereby collapsing the array even though all the disks are still functional.
A rebuild initiated with the member disks incorrectly ordered or incorrect parameters, will write bad parity across existing good data and create one of the few possible ways to render an otherwise recoverable array non-recoverable.
Following a failed rebuild operation, the first correct course of action is no action. Typically, most “degraded” arrays that crashed while rebuilding, are also very recoverable, but nearly every “just let me try one more time”, “fix this problem”, etc., decreases the likelihood of recovery. Each forced retry, each attempted fix/recovery, and each attempt to get the array operational again increases the risk of overwriting the parity & file system information required to perform a successful recovery operation. Therefore, power down the array and do not touch it.
Taking a few careful steps now will protect the data that exists on the drives so when it comes time to recover everything, it will be cleaner & faster.
Cleanly shut down and stop all use of the NAS or server. It will do nothing for itself; every hour the system remains running degraded is an additional time to lose a hard disk.
Before you remove anything, take a marker and label each hard disk with the location (bay or slot number). In order to restore the RAID configuration, knowing which disk was where is essential. Mark them now so this is documented.
Send all members (even if one was "lost") and document everything about what happened, as well as which member went bad first and any repair attempts you may have made prior to contacting us. We require a full view of the situation.
Our goal is to rebuild the RAID configuration on a completely different set of disks than those originally used. All work will be performed from duplicate copies of your data (read-only), here at our facility.
We check the integrity of each disk (both good and the ones that failed) and determine which ones have physical problems vs. the good ones. This way, you'll know exactly which disks can be restored and you'll receive a fixed, written quote before any cost is incurred by you.
Each drive is cloned separately via a write-blocking link, never touching the original. A faulty mechanical component is fixed first, as little as needed to create an image of the disk.
The stripe size, the order of the disks and the parity scheme are reconstructed in software from the images. This creates the virtual array offline, away from the hardware controllers, with no chance for failure on the physical disks.
The file system and data are pulled from the virtual array and separated into two groups: good files prior to rebuilding and bad files written after rebuilding. You receive a complete list of items prior to your final decision.
Most common questions asked after a rebuild attempt was unsuccessful.
No. In a degraded array there is no longer any redundancy, thus performing another rebuild will increase the probability of experiencing a catastrophic read failure or having a second drive fail completely, in addition to potentially damaging recoverable data by writing bad parity. Stop, and let the drives be imaged first.
Most likely not. We are usually seeing RAID jobs where a degraded array had crashed during a rebuild; we can recover this through imaging every disk to then virtually rebuild them, provided no-one has brought the system back online, or run any repair utilities on it.
To a great extent, yes. The rebuild process relies upon knowledge of the original drive order, the striping block size (stripe size) and how the disks were laid out (parity). Before you remove all your drives, identify which bay they came from; if the order is lost we can still figure it out, although it will take longer.
Yes, please send us everything. Often even failed members have enough usable information for us to perform a clean rebuild and also image whatever data we can read from them. Attempting to recover an array using only a portion of the total drives involved is a lot harder.
Pricing for RAID/NAS/server recovery starts from £500 + VAT and varies with the amount of drives you have, and based on how many of those drives need to be recovered. We provide you with a fixed price prior to doing any chargeable work. There is a complimentary diagnostic before we start any billing.
Do not attempt to rebuild again or bring the array online again. Turn power off, and label all the drives by their order, and send all of them to us. We will use read only imaging techniques to image each disk, and then we will virtually rebuild it without ever touching your original disks.