Some stuff » hard disk recovery https://blog.yhuang.org here. Wed, 27 Aug 2025 08:50:58 +0000 en hourly 1 http://wordpress.org/?v=3.1.1 Useful information (Appendix) https://blog.yhuang.org/?p=47 https://blog.yhuang.org/?p=47#comments Mon, 08 Jan 2007 03:24:04 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=47 This is part of the hard disk recovery documentation.

Appendix.

Here are all the tools that made an arguably irreplaceable contribution in the recovery:


Here are some additional links that I looked at, with more information:

For the record, here are the two external disks I have with their ext2 partitions’ Superblocks shown:

100GB disk, just one ext2 partition, starts at sector 63.

160GB disk, there is a 20GB FAT32 partition, and the second ext2 partition starts at sector 41945715. This is the partition whose backup superblocks are all off by +1KB (but are otherwise correct), and hence has an illegal formatting.

And here are the ddrescue log files, from the first run and the second run.

Back to the beginning.

]]> https://blog.yhuang.org/?feed=rss2&p=47 7 Today I can go no farther (part 6) https://blog.yhuang.org/?p=46 https://blog.yhuang.org/?p=46#comments Sun, 31 Dec 2006 03:53:21 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=46 This is part of the hard disk recovery documentation.

Part 6.


Today I can go no farther (so I stopped)

The last days of this project are spent on two tiring tasks that do not gain me very much, but must be done to carry this project to its logical conclusion. One of these is to decrypt a few very small, but important NTFS-encrypted files. The other is to wring the last readable bits out of the broken Seagate drive by splitting all the error regions to isolate the unreadable regions as much as possible. These can proceed in parallel.

I set Knoppix to slave away on the broken Seagate drive using ddrescue again. This ended up taking more than one day constantly running and made the disk really hot. I had to manually intervene from time to time just so this will get somewhere. This is where I wished ddrescue was smarter about skipping around unreadable regions.

The other task was more interesting from a learning perspective. When a file is to be encrypted by NTFS, it generates a file-specific symmetric “file encryption key” and stores the key with the file. Of course the key isn’t stored in plaintext but is encrypted using the user’s public key and is only decodeable by the user’s private key. Normally the user should export the private key into a .pfx certificate file and store it on a smart card or USB pen drive or something, and more importantly, remove it from the certificate store on Windows. I did make a copy of the private key off the computer. I could just re-import the certificate, and then decrypting the data would be trivial. But the external storage device that holds the private key isn’t with me during the break, unfortunately. On the other hand, I didn’t bother to remove the private key from the computer either, because I wasn’t really serious about this particular piece of data, just toying around with the NTFS encryption feature, mostly.

That made the encrypted file potentially recoverable without delay. Windows doesn’t store the user’s private key in plaintext either, of course. From what I understand, the user’s private key is encrypted somehow with user’s password hash, then maybe it is encrypted again with a master key belonging to the system. Somehow, the system’s master key is the weak link. I think it is stored in the SAM database of the system registry, which is in %SYSTEMROOT%\system32\config, though I may be mistaken on this point. The user’s various keys are stored in %USERPROFILE%\Application Data\Microsoft\Protect and %USERPROFILE%\Application Data\Microsoft\Crypto. Here is another explanation of this.

To make a long story short, if all of these files are available, and the user’s password is known, then it is possible to recover the encrypted files directly from the drive without an explicitly exported private key (hence encryption with these files around is ultimately pointless).

I ended up using the Elcomsoft Advanced EFS Data Recovery tool (AEFSDR). They have a trial version, too. Unfortunately it’s crippleware, unlike Mount Image Pro. All I can say is, if you really want to decrypt files, you find a way around the “cripple” in crippleware. Enough said.

For the record, AEFSDR is pretty good and fast. However, I am also not completely satisfied by it. I haven’t found any bugs, but the problem with it is its incompleteness. I don’t know what file system access method it uses, but it cannot see any removable drives, including removable USB drives or any mounted virtual disks (e.g. by Mount Image Pro). This is severely limiting, because encrypted files are not easily copied (Windows says Permission Denied), and I basically ran into a wall because of this. Please, AEFSDR, go work on this some more. You are selling a recovery tool, so your users are most likely in desperate straits with limited resources… they don’t want to deal with your programming kludges.

The way around it is to use NTBACKUP (part of Windows), which can archive encrypted files to a backup image, and then you can restore these files onto a permanent physical disk. After also copying the security related files from Windows to a permanent physical disk, AEFSDR decrypts all files successfully.

RELOADING THE DDRESCUE IMAGE

After I got as much as I am ever going to get out of the Seagate disk using ddrescue (I got another few tens of megabytes out of the disk), I retraced the steps on the new image. This time, the image loads as a valid NTFS partition and Windows will read it (apparently the last few bits recovered was pretty important). However, the Windows directory is still not readable, although almost all other directories are directly readable. NFI also works on the image — this is very good, as it means I can finally profile what I lost down to the last sector. I also ran chkdsk on it again — fewer errors to correct this time, but hopefully fewer zero-filled data issues to deal with at the end of the day. This chkdsk’d image is now the gold recovery image. I backed it up immediately.

That is the end of it. The story has a happy ending, but what a giant waste of time. Back up your data.

Lessons today:

  • AEFSDR is nice. But I wouldn’t buy it, either.
  • Remove your private keys from the computer.
  • Near total data recovery is very possible from an apparently dead drive, but a delicate procedure needs to be followed.
  • The right tools are very important, and a complete disk recovery toolchain simply does not exist — you are left to find your own pieces.

On to the Appendix.

]]> https://blog.yhuang.org/?feed=rss2&p=46 2 I rammed my head against the wall, and all was clear (part 4) https://blog.yhuang.org/?p=44 https://blog.yhuang.org/?p=44#comments Wed, 27 Dec 2006 21:23:44 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=44 This is part of the hard disk recovery documentation.

Part 4.


I rammed my head against the wall (of absurdity that is ext2), and all was clear

The obvious need to fork off another hard disk recovery project from a hard disk recovery project was just too pathetic to think about so I kicked this aside for a day. Today I started back from the beginning, this time researching the ext2 filesystem. The documentation (that is within reach of Google) for this file system is just poor. Whoever is responsible for documenting this part of Linux should be flogged, or at least made to go home and do it over. Yeah, I can go look at the source code (which I did), but I’m sorry, that does not constitute proper documentation.



What I gathered:

The ext2 physical structure consists of a basic unit called a block, whose size in bytes can be set at formatting time. Contiguous blocks are organized into groups, and contiguous groups make up the partition. The groups are of equal size, each containing by default 32768 blocks (on my partition a block is 4KB, so I guess a group is 128MB).

The next part is where I depart from the random wrong stuff strewn around the internet. I know they are wrong because I actually looked at the disk, but it caused me a lot of headache. The first group (Group 0) contains a 1KB filler, which some people like to call the “boot block” for unknown reasons (so far as I can tell, it doesn’t boot anything and it isn’t a block in size), followed by the 1KB Superblock (also not a block in size), which is like a filesystem header. It carries a bunch of file system parameters, and an identifying two-byte sequence called the “magic number.” At 4KB (for my block size) into Group 0 begins the Group Descriptor Table, which is also a filesystem header. The Group Descriptor Table contains 32-byte entries, one entry for each group on the disk, and each entry telling the location of the block bitmap (i.e. allocation table for blocks) and the inode bitmap (i.e. allocation table for inodes) belonging to that group.

The subsequent groups all have the same Group Descriptor Table at the same position in the group as in Group 0, and some of the groups also have a copy of the 1KB Superblock at its front (not at the 1KB position). The last part is important. The first two Superblocks are not 32768 blocks apart.

That is the gist of it. I didn’t bother with the block and inode allocation strategies because those aren’t important to me.

So when Knoppix said the Group Descriptor Table was corrupt, it means one or more of entries are pointing to the wrong places. Also when it says could not find the Superblock due to a “bad magic number,” it most likely means the Superblock copies in other groups are all overwritten, or, not where they are expected to be found, so they or the Group Descriptor Tables can’t be used to correct the Group 0 ones.

To find out what happened exactly, a disk editor was needed. I used to know a pretty good one for DOS, but I lost it. However, Windows Support Tools for Advanced Users has a Windows tool (used to be in the NT resource kit) called DiskProbe (dskprobe.exe) that does the trick. It’s a bit annoying that disk positions must be counted in units of sectors (512 B), but oh well. Also the Linux tool called fsstat was used, which dumps information read from the Superblock and Group Descriptor Table out one group at a time, whatever they contain (corrupt or not).

What I found, from fsstat, and from comparing with another disk formatted similarly with ext2, is all the Superblock copies in the groups that should have them are there, but offset by 1KB. How this happened I do not know. Seems like a bug of mkfs to me (it has to be mkfs because where the Superblock is supposed to be resides prior data). As for the source of the problem, the Group Descriptor Table stored in Group 0 has one entry in which 4 bytes were overwritten with 0x7FFFFFFF. How that happened I also do not know for sure, but it is almost certainly the fault of ext2ifs, since that is the only thing to touch the partition between it working and not working. But as 4 bytes appears to be the extent of the corruption, and the backup Group Descriptor Tables are perfectly good (they just can’t be found by silly Knoppix), I just copied four bytes by hand in DiskProbe, and bam, the ext2 file system was mountable again by Knoppix.

The absurdity is that not only did a presumably illegal ext2 file system get written onto the disk, but also none of the ext2 fs check tools were able to find where the correct Superblocks were, even though they were just 1KB away, and furthermore, gpart, the Linux program that is supposed to guess what partitions are on the disk, could not even identify this partition as ext2, despite the Group 0 Superblock being intact and all the other structures being obviously ext2. Whoever wrote these parts of Linux should be flogged, or at least made to go home and write documentation instead.

Lessons today:

  • The backup littering “strategy” of ext2 is only better than nothing… it is way too simple-minded and needs way too much hand-holding.
  • Not one piece of software can be trusted to do the right thing.

On to Part 5.

]]> https://blog.yhuang.org/?feed=rss2&p=44 0 Today I became suspicious of everything (part 3) https://blog.yhuang.org/?p=38 https://blog.yhuang.org/?p=38#comments Mon, 25 Dec 2006 17:06:04 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=38 This is part of the hard disk recovery documentation.

Part 3.


Today I became suspicious of (the ext2ifs driver, the mkfs command, the USB enclosure, and basically) everything

On Christmas morning Santa Claus had not granted my wish: ddrescue was still running, but the image file had not been timestamped any more recently than when I left it, and the damaged drive had spun down by itself. dmesg revealed a syslog message “too many IO errors” or something like that, which had caused Linux to give up on reading from the damaged drive. I was very frustrated because, well let’s see, I had expected the disk imaging to make good progress, but instead… I must suffer a reboot and the induced indefinite re-churning of the drive, with even more data loss! What.

Attempting to reboot, I got impatient and tried the Dell Diagnostic partition again. To my dismay, the Dell Diagnostic partition had become unreadable, a sure sign that the disk’s failures were worsening, adding more urgency to the recovery mission. With no viable second option, I booted Knoppix again and resigned myself to the disk churning. Finally I was in again and started ddrescue back up at about 4000MB, to skip the errors at 3200MB.

/media/sda2# ddrescue -B -n -i4000M /dev/hda rescue.image rescue.log

At some point I had to get a replacement hard drive. 80GB seemed to be the right price point, and somehow I narrowed it down to just two “choices”: a Seagate Momentous 5400.3 (with Perpendicular Recording Technology) believe it or not, and another one by Fujitsu, so it wasn’t much of a choice. The Seagate had already received reviews noting bad sectors (!) and I really did not want another one of those. I also rang Dell to see if the parts would be interchangeable (mostly on the question of the CD drive). Unfortunately, Dell would not give me a straight answer, but after insisting on getting my personal information, they told me “your warranty has expired,” “but you have unlimited lifetime phone support,” which turned out to be a person on the other end putting me on hold for minutes at a time every time I asked a question to go read a manual and coming back with some inane suggestion. For example, I was told to repeat things I said I already tried, in particular to go into the Dell Diagnostic partition which of course was already dead. His eventual diagnosis was, “it has been determined that, sir, you have a hardware problem, not a software problem, and due to your warranty has already expired, you may get a new drive from anywhere, just make sure to get a 40GB IDE disk” (presumably to match the original). So I wrote off the wasted hour on that call as sunk cost, and went ahead with the Fujitsu drive.

It continued to stick briefly at roughly 1GB intervals of data transfered, sometimes less. Sometimes I would manually interrupt the process and re-start it at a later point. This ddrescue allows with no problem because it uses its log intelligently, to patch together various chunks of recovered data, such that validly transferred data was never reread and interruptions were not costly to the recovered image. Wishing to avoid a repeat of the inextricable-3200MB-error from Sunday night, I interrupted ddrescue whenever it seemed to be stuck long. I noted the following sticking-points:

3277MiB
9798MiB
10415MiB
10917MiB
11906MiB
13885MiB
24499MiB
25890MiB
27394MiB
27851MiB
31280MiB
32799MiB
33748MiB
35065MiB
36954MiB

The data corruption pattern covers the entire disk — so far as there is any physical correspondence — but affects only a small percentage of disk’s data. In fact, the recovery to error ratio was at least 100:1. I eventually transferred about 38GB of raw bits out of the 40GB disk.

By the way, the -n option demands “no error splitting” meaning that a 64K-chunk of data with any read problems is marked as wholly erroneous (or “/” in the log file… good data is “+” in the log file). After the runs with -n, I did a run with error splitting on a small portion on the disk, forcing ddrescue to analyze only the 64K-chunks with errors on the original run. With this method, I recovered yet more data (“/” became mostly “+” and some “-”).

TRYING THE RAW IMAGE UNDER WINDOWS

Before attacking more chunks with error-splitting, I decided to examine the current state of the disk image. There would be no point to try very hard on the bad parts if the useful files were already contained in the “+” regions. Just from the log file, I was pleased to find the first 5K intact (this includes the first sector holding the MBR and partition table), then many errors in the front of the drive which were in the Dell Diagnostic partition, followed by several hundred MB intact, which would include the boot sector of the NTFS partition and the front of its master file table hopefully. A chunk of the end of the drive was also intact, which should contain the backup NTFS boot sector. Things were looking much better.

So I next planned to mount the image under Windows, since many NTFS analysis tools run only on Windows. To do this, I needed a filesystem driver to read ext2 under Windows. There are a few, but the “ext2 Installable File System for Windows” or ext2ifs boasts Kernel-mode extension of the Windows file system that “is indeed comparable to Windows NT’s native file system drivers“. Not really, as the implementation is missing some behavior one would expect from Windows but it does behave in such a way that most of the usual higher-level filesystem manipulations can be done directly on the ext2 volume, including getting real drive letters and the ability to perform file manipulation directly (without requiring a specialized file manager).

First I wanted to make a copy of the image file, for two reasons: to have a copy to work with without endangering the recovered data, and to put the original back into ddrescue to churn away at erroneous chunks according to which chunks turn out to be important. In retrospect I could have made this copy under Linux, but since I had ext2 already mounted in Windows, I just made the copy under Windows. While it was doing this copying, guess what, the source drive (the external USB drive) became unreadable. Uh……

I took the drive with the 38GB of good data back into Knoppix Linux, and Linux could not mount it either. Uh……

# dmesg | tail

“Corrupt group descriptor: bad block for inode bitmap” etc. etc. Uh……

What did the ext2ifs driver do? There is not a single reason it should have touched the group descriptor table.

# e2fsck /dev/sda2

“e2fsck: Bad magic number in super-block while trying to open /dev/sda2
The superblock could not be read or does not describe a correct ext2 filesystem.” etc. etc. Uh……

What did mkfs.ext2 do? There are backups of these file system parameters, why are they all bad now?

What did the USB enclosure do? It didn’t barf all over the disk, did it?

This completely goes against the original intention of having a good copy of the data! And at this point I need to fork a sub-project to work on recovering the ext2 volume, because I don’t think I can read all 38GB of the already-recovered data back out of the dead Seagate again. Argh!! I called it a night.

Lessons today:

  • I shouldn’t fork the project, I should just bork it altogether. The gods clearly aren’t working with me.
  • Of course I should keep working at this. I just need to use proven technology when making a crucial file copy.
  • ddrescue is good about skipping over good areas of the disk already scanned, but isn’t intelligent about scanning the bad areas of the disk.
  • Dell support is truly useless.

On to Part 4.

]]> https://blog.yhuang.org/?feed=rss2&p=38 0 I tried a whole bunch of things, and all that (part 2) https://blog.yhuang.org/?p=37 https://blog.yhuang.org/?p=37#comments Sun, 24 Dec 2006 08:18:13 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=37 This is part of the hard disk recovery documentation.

Part 2.


I tried a whole bunch of things (most that didn’t work), and all that.

What is there to do? Data recovery services that run into the thousands of dollars can probably get most of the data back — they have a track record of that. My data isn’t worth nearly that much, sad to say. But I don’t feel like abandoning perfectly good data, either. Yes, there is probably McNorton ViralGhostSpy or whatever this bloatware is called these days; I don’t know… I prefer more flexibility so I’ll take the trouble to proceed with free or freely available tools.

Since the Seagate disk is too dead to figure out what is on itself, I need to get its contents off onto another medium. I need two things: some means of getting the only computer that can read the disk as is (the laptop) into a working state, and some means of direct disk copying from the laptop. I have an enclosured USB drive: 160GB. Let’s work with that. To protect the original broken Seagate, I left it out of the laptop for the time being.

Many a self-proclaimed miracle recovery boot disks/discs conceivably take care of both tasks. A quick search turns up a boot floppy software called “EaseUs disk copy” using a self-booting “freedos” but it was useless because it would like to copy within a computer (from master IDE drive to slave). Into the trash it goes. There were a few others like this, all useless.

Next up, I’ve got some Windows-based disk editor tool. Maybe I can install Windows on this external USB hard disk so I can boot the laptop? No. Evidently it isn’t supported. Windows installed fine on the external USB drive (all files are copied), but on the first boot BSOD’d. Some say there is a way to hack the Windows CD to make this work. Well I don’t have that much time to waste, so forget that.

Next up, Knoppix version 5 (Linux Knoppix 2.6.17 #4 SMP PREEMPT, i686 GNU/Linux). I had some difficulty mounting the external USB hard drive, but finally by booting from CD, then attaching the external drive after startup, Knoppix found it and stuck it under /dev/sdaN (for N in 1,2,etc.). The drive had a 20GB FAT32 partition already (where I attempted to install Windows before), along with an empty 120GB partition, so the empty partition was at /dev/sda2.

These days, the favorite recovery tool under Knoppix seems to be the ignore-on-read-error disk copy tool called ddrescue (not the Knoppix-included dd_rescue). I believe the difference is that ddrescue is much more automatic in dealing with the log file — this is important. Both will copy a disk’s contents into a disk image file. ddrescue needs to be compiled though; fortunately compilation was clean.

The choice of filesystem for /dev/sda2, which would hold the disk image, was limited. The plan was to eventually read the data on a Windows computer, which would be easiest with FAT32 or NTFS. However, NTFS write-support on Knoppix is of unknown reliability (– why? I don’t know). I tried FAT32, using mkdosfs to format because Windows refuses to format a FAT32 volume larger than 32GB. Then I even began copying data before realizing, to my chagrin, that FAT32 has another limit: 4GB on the file size. That left ext2, and its default block-size of 4K was fine (block-size under 2K would be a problem due to file size limitation), so something like:

# mkfs.ext2 /dev/sda2
# mount -t ext2 /dev/sda2 /media/sda2

Progress.

Now, computer off. Broken Seagate goes back in. Boot. And… no boot. The boot sequence now stuck at two points where it wanted to read the Seagate drive (on /dev/hda): First, it attempted to find a (nonexistent) CD on /dev/hda, resulting in churning the hard drive for a few seconds (damaging my data further, I’m sure!) Next, at the auto-detection stage to create /etc/fstab, Knoppix repeatedly tried to read the drive, and apparently would go no further. Online sources are rife with suggestions of passing the boot options “nofstab” and “noswap” to the kernel to solve this problem. Wrong. With these options set, it still had the same problem at the auto-creation of /etc/fstab. The solution turned out to be nail-biting patience: After booting and allowing the auto-detection to proceed for 10+ minutes of now-familiar churn-churn-churn on the broken Seagate, heating it up and losing me more data, it eventually did go on and boot. Um… whoever is responsible for this boot code should be flogged or at least, made to go home and work on it some more. I can think of zero (0) reasons why this particular process warrants persisting through multiple disk read errors.

Now, /dev/hda wouldn’t mount (of course), but no matter. Here we have:

/media/sda2# ddrescue -B -n /dev/hda rescue.image rescue.log

ddrescue proceeded through the evening, surprising me by writing quite a bit of good data onto the disk image, even as it sticks briefly at various corrupted points, but then got bogged down at around 3200MB. I left it to its own devices at 3AM.

Lessons today:

  • Knoppix is genius. Why is there no Live-Windows-on-CD? Oh, I know why.
  • Knoppix is semi-retarded for disk recovery because it fights with a broken hard disk on boot.
  • FAT32 has a 4GB file-size limit even if you manage to format the volume to be rather large.
  • ddrescue is a great tool to have.

On to Part 3.

]]> https://blog.yhuang.org/?feed=rss2&p=37 1 Today I became suspicious of Seagate products (part 1) https://blog.yhuang.org/?p=36 https://blog.yhuang.org/?p=36#comments Thu, 21 Dec 2006 23:43:09 +0000 admin http://scripts.mit.edu/~zong/wpress/?p=36 This is part of the hard disk recovery documentation.

Part 1.


Today I became suspicious of Seagate products (and my fortune in general)

Windows XP was running, and programs were being used. The disk was probably being accessed for memory cache. I noticed the drive making repetitive noises, spinning down and then spinning up, and the machine became unusable. I power-cycled the machine, and it was “NTOSKRNL.EXE is missing or corrupt.” Bad news.

To diagnose the problem, I booted the machine into “Dell diagnostic partition”, located at the front of the hard disk. This partition booted with no problem. Good? I used the “disk diagnostic” tool, which passed basic disk parameter checks (size of disk, for example), but upon reading the disk soon found read errors on some sectors. I started the same test at a later location to skip the damaged sectors, and passed a long error-free zone of the disk, but then encountered more erroneous sectors. After this, I tried the “disk confidence test” tool, which reads only small portions of the disk at a time, as a representative sampling of disk integrity. The test was proceding well, but I halted it for time reasons.

Next thing to do was to boot off the Windows XP CD, hoping to use its Recovery Console to see if the file system is still okay, and maybe try to fix the file system if the problem isn’t too serious. At this point however, my CD drive made a loud clicking noise, and I found the spindle plastic at the axis of the CD had come off the drive. Two malfunctions on the same day? Woe is me. What’s wrong with this picture — I don’t believe I killed any small animals today (or on any day really)!

So I pushed the spindle plastic back on, tried again. No luck. I conclude that the CD-ROM, my only other means of mass-input, is broken just when I need it quite much. In my last attempt to fix anything today, I tried white-out as glue on the plastic part. This worked. Unfortunately, the CD drive vibrates far more than before, due to its spindle being now slightly off-kilter. This isn’t going to last for very long…

So, in goes the XP CD and Recovery Console comes up, but the OS installation can’t even be found on the disk and there is no C: drive to inspect. Ouch. This is serious. One screw off and out pops the hard disk. Hoho! It’s a Seagate Momentus ST94811A, a so-called “reliable” drive. This machine is barely two years old! Woe be to Seagate which made the drive. Woe be to Seagate Momentus which is designed to run way too hot under the palm for a laptop hard disk anyway. That should have tipped me off. Hrmmmm.

Lessons today:

  • When hard disk seems to churn more than usual paging (as in defragmentation-like churn), it may die soon.
  • When your browser repeatedly crashes for no apparent reason after paging in from disk, yet your RAM is verifiably good, your disk may die soon.
  • White-out is reasonably good plastic glue.
  • Dell diagnostic partition takes up almost 50MB of disk space.
  • Please choose a Seagate Momentus if you would like to burn your palm and lap.
  • If there is some data you would rather not lose at this very moment, back up now.

On to Part 2.

]]> https://blog.yhuang.org/?feed=rss2&p=36 0