Guide · RAID & NAS

RAID won't rebuild? Stop here first.

The single most common reason we see users lose their data forever is a RAID rebuild failure. Users are usually inclined to simply remove the bad disk and rebuild again. This is essentially the same as destroying the RAID. Here's what to do next:

Free 48-hour diagnostic
Handled in-house
Only pay if we recover your data · most jobs
// in short

Don't force a rebuild.

If you are rebuilding to a faulty disk or building in the wrong order, you could be overwriting your data permanently. Immediately stop and shut down the system. Don't change anything until you have had this evaluated.

Rebuild
= top risk
Label
Bay order
Drives only
Not the box
Read-only
We image first
// what's happened

A degraded array is running without a net.

There is no shortage of RAID emergencies where a rebuild stalls or was stopped (or even worse) leaves the array unable to mount. Knowing how RAID 5 works may help understand why rebuilding an array after it fails is so dangerous. Your RAID stores each file piece across multiple drives, but also includes a rotating block of parity data for each stripe. When a RAID loses a drive, parity lets it rebuild the missing data from the surviving drives, but no redundant information is left. In other words, once one drive is dead, the array has no safety net. It is now degraded and runs with no redundancy. The rebuild process is by far the most stressful operation a RAID can perform. Thus, when a RAID is running with no safety nets during a rebuild, a second problem will likely appear.

// why rebuilds fail

Why the rebuild fell over.

Nearly all rebuild failures boil down to one of these issues and they are all caused by the fact that the RAID is working flat-out with no fault tolerance remaining.

01

A second drive fails

The rebuild will look at every sector of every available (surviving) disk. Because those disks have been used as long and have worn out equally as the dead disk, the constant usage puts enough strain on a marginal disk causing it to fail, and now two disks are down.

02

A bad sector (URE)

In order to build the stripe again, the controller has to read all the remaining data. If the controller hits an unreadable sector without any redundant sectors left to help fill in missing sectors, it cannot reconstruct the stripe; therefore, it will abort the rebuild.

03

A “phantom” drop

The controller may kick a consumer (healthy) disk out of the array because it took too much time reading through a sector that was slower than normal; thereby collapsing the array even though all the disks are still functional.

04

Wrong order or config

A rebuild initiated with the member disks incorrectly ordered or incorrect parameters, will write bad parity across existing good data and create one of the few possible ways to render an otherwise recoverable array non-recoverable.

// the one thing that matters

The most important step is to stop.

Following a failed rebuild operation, the first correct course of action is no action. Typically, most “degraded” arrays that crashed while rebuilding, are also very recoverable, but nearly every “just let me try one more time”, “fix this problem”, etc., decreases the likelihood of recovery. Each forced retry, each attempted fix/recovery, and each attempt to get the array operational again increases the risk of overwriting the parity & file system information required to perform a successful recovery operation. Therefore, power down the array and do not touch it.

×Do not continue re-running the rebuild: with no redundancy left, you will have a second drive at risk for failure or a fatal read error as soon as you try the next rebuild.
×Do not force the array online or mount it read/write and do not run chkdsk, fsck or any of the NAS repair tools.
×Do not re-initialise, re-format or “reset” the array: this is the fastest way to lose all of your data.
×Do not change the order or swap out drives to see which ones work: the original order is part of what makes recovery possible.
// what to do

Label everything, then stop.

Taking a few careful steps now will protect the data that exists on the drives so when it comes time to recover everything, it will be cleaner & faster.

Do

Power the array down

Cleanly shut down and stop all use of the NAS or server. It will do nothing for itself; every hour the system remains running degraded is an additional time to lose a hard disk.

Do

Label the drives in order

Before you remove anything, take a marker and label each hard disk with the location (bay or slot number). In order to restore the RAID configuration, knowing which disk was where is essential. Mark them now so this is documented.

Do

Send the whole set in

Send all members (even if one was "lost") and document everything about what happened, as well as which member went bad first and any repair attempts you may have made prior to contacting us. We require a full view of the situation.

// how we recover it

How a failed array is recovered.

Our goal is to rebuild the RAID configuration on a completely different set of disks than those originally used. All work will be performed from duplicate copies of your data (read-only), here at our facility.

01

Assess every drive

We check the integrity of each disk (both good and the ones that failed) and determine which ones have physical problems vs. the good ones. This way, you'll know exactly which disks can be restored and you'll receive a fixed, written quote before any cost is incurred by you.

02

Image each drive read-only

Each drive is cloned separately via a write-blocking link, never touching the original. A faulty mechanical component is fixed first, as little as needed to create an image of the disk.

03

Rebuild the array virtually

The stripe size, the order of the disks and the parity scheme are reconstructed in software from the images. This creates the virtual array offline, away from the hardware controllers, with no chance for failure on the physical disks.

04

Extract, verify, file list

The file system and data are pulled from the virtual array and separated into two groups: good files prior to rebuilding and bad files written after rebuilding. You receive a complete list of items prior to your final decision.

// faq

Common questions.

Most common questions asked after a rebuild attempt was unsuccessful.

No. In a degraded array there is no longer any redundancy, thus performing another rebuild will increase the probability of experiencing a catastrophic read failure or having a second drive fail completely, in addition to potentially damaging recoverable data by writing bad parity. Stop, and let the drives be imaged first.

Most likely not. We are usually seeing RAID jobs where a degraded array had crashed during a rebuild; we can recover this through imaging every disk to then virtually rebuild them, provided no-one has brought the system back online, or run any repair utilities on it.

To a great extent, yes. The rebuild process relies upon knowledge of the original drive order, the striping block size (stripe size) and how the disks were laid out (parity). Before you remove all your drives, identify which bay they came from; if the order is lost we can still figure it out, although it will take longer.

Yes, please send us everything. Often even failed members have enough usable information for us to perform a clean rebuild and also image whatever data we can read from them. Attempting to recover an array using only a portion of the total drives involved is a lot harder.

Pricing for RAID/NAS/server recovery starts from £500 + VAT and varies with the amount of drives you have, and based on how many of those drives need to be recovered. We provide you with a fixed price prior to doing any chargeable work. There is a complimentary diagnostic before we start any billing.

// rebuild failed?

Rebuild failed? Power it down and stop.

Do not attempt to rebuild again or bring the array online again. Turn power off, and label all the drives by their order, and send all of them to us. We will use read only imaging techniques to image each disk, and then we will virtually rebuild it without ever touching your original disks.

Call us — 028 9002 0144
Mon–Fri · 9am–5:30pm · No fix, no fee
Start a free diagnostic →
028 9002 0144