The other task was more interesting from a learning perspective. When a file is to be encrypted by NTFS, it generates a file-specific symmetric “file encryption key” and stores the key with the file. Of course the key isn’t stored in plaintext but is encrypted using the user’s public key and is only decodeable by the user’s private key. Normally the user should export the private key into a .pfx certificate file and store it on a smart card or USB pen drive or something, and more importantly, remove it from the certificate store on Windows. I did make a copy of the private key off the computer. I could just re-import the certificate, and then decrypting the data would be trivial. But the external storage device that holds the private key isn’t with me during the break, unfortunately. On the other hand, I didn’t bother to remove the private key from the computer either, because I wasn’t really serious about this particular piece of data, just toying around with the NTFS encryption feature, mostly.

That made the encrypted file potentially recoverable without delay. Windows doesn’t store the user’s private key in plaintext either, of course. From what I understand, the user’s private key is encrypted somehow with user’s password hash, then maybe it is encrypted again with a master key belonging to the system. Somehow, the system’s master key is the weak link. I think it is stored in the SAM database of the system registry, which is in %SYSTEMROOT%\system32\config, though I may be mistaken on this point. The user’s various keys are stored in %USERPROFILE%\Application Data\Microsoft\Protect and %USERPROFILE%\Application Data\Microsoft\Crypto. Here is another explanation of this.

To make a long story short, if all of these files are available, and the user’s password is known, then it is possible to recover the encrypted files directly from the drive without an explicitly exported private key (hence encryption with these files around is ultimately pointless).

I ended up using the Elcomsoft Advanced EFS Data Recovery tool (AEFSDR). They have a trial version, too. Unfortunately it’s crippleware, unlike Mount Image Pro. All I can say is, if you really want to decrypt files, you find a way around the “cripple” in crippleware. Enough said.

For the record, AEFSDR is pretty good and fast. However, I am also not completely satisfied by it. I haven’t found any bugs, but the problem with it is its incompleteness. I don’t know what file system access method it uses, but it cannot see any removable drives, including removable USB drives or any mounted virtual disks (e.g. by Mount Image Pro). This is severely limiting, because encrypted files are not easily copied (Windows says Permission Denied), and I basically ran into a wall because of this. Please, AEFSDR, go work on this some more. You are selling a recovery tool, so your users are most likely in desperate straits with limited resources… they don’t want to deal with your programming kludges.

The way around it is to use NTBACKUP (part of Windows), which can archive encrypted files to a backup image, and then you can restore these files onto a permanent physical disk. After also copying the security related files from Windows to a permanent physical disk, AEFSDR decrypts all files successfully.

RELOADING THE DDRESCUE IMAGE

After I got as much as I am ever going to get out of the Seagate disk using ddrescue (I got another few tens of megabytes out of the disk), I retraced the steps on the new image. This time, the image loads as a valid NTFS partition and Windows will read it (apparently the last few bits recovered was pretty important). However, the Windows directory is still not readable, although almost all other directories are directly readable. NFI also works on the image — this is very good, as it means I can finally profile what I lost down to the last sector. I also ran chkdsk on it again — fewer errors to correct this time, but hopefully fewer zero-filled data issues to deal with at the end of the day. This chkdsk’d image is now the gold recovery image. I backed it up immediately.

Then I took the copy of the image under Windows. This way I didn’t care what the ext2ifs driver wanted to do. But this time it didn’t mangle the partition (probably because this ext2 partition actually has a legal physical formatting!).

Finally I can run NTFS tools on the broken disk, but I need to mount the (broken) NTFS image first. There is a free tool, FileDisk by Bo Branten, to do this. Unfortunately, it gives a drive letter to the whole disk, instead of to the partitions. This is nearly impossible to work with because (1) there are 63 sectors of MBR, partition table, and filler at the front of the disk, and (2) there is the Dell diagnostic partition. And FileDisk doesn’t virtualize a physical disk device, which would be the more correct metaphor.

A way around this is to use VMWare, and virtualize the disk image as a disk. But no, that’s too much trouble… although, after this incident, I’m really considering putting a VMWare image with Windows preinstalled onto a DVD or something — that would be the complement to Knoppix.

A somewhat more advanced tool called Mount Image Pro (MIP) exists that does what I want. It isn’t free (*), but who cares when there is a one-month trial? (For the record, I’m not entirely happy with this thing, either. It works, but almost every other time it fails to load the system driver and requires a reboot and retry.)

With MIP, the NTFS partition comes up and is mounted under K:, but it is completely broken as expected. The file system cannot be read normally by Windows, nor would the diagnostic tool NFI (part of the OEM Support Tools) work on it. NFI relates what files a sector contains and what sectors a file resides in.

The next step is then to run chkdsk:

> chkdsk /v K:

Very specific errors relating to the file table are immediately detected and chkdsk says it cannot continue in the (default) read-only mode.

So let’s try

> chkdsk /v /r K:

Same complaint by chkdsk. What? It turns out MIP (at least the GUI) only mounts images in read-only mode, but it doesn’t specify this anywhere! Nor does it give an indication of how to get around this. Wow great.

After poking around, I noticed there is a command-line interface to MIP, too, and it is there that you can specify mounting in “read-write” mode, instead of “write-block” mode. (I would also call “write-block” mode “read-only” mode, but that’s just me, so what do I know!) Moving on:

> mip mount rescue.image /rw /p:2 /l:K

Finally, the second partition in the disk image is mounted on the drive letter K: (hence the last two parameters). Unfortunately, MIP screws up again and says there is only “1″ partition, and the drive letter K is associated with the first, Dell diagnostic, partition… even though it clearly displays two partitions and that the drive letter K is in reality associated with the second partition. No matter, it still works, so I’ll leave the MIP people unflogged. But they should still go home and fix these.

Trying chkdsk again and it slaves away through a 5-step process, dumping output on all the files and directories on which it detected problems. The disk image is broken enough that chkdsk had to kill most of the NTFS file permission. After about 2 or 3 hours, K: is miraculously readable again in Windows, with the basic directory structure at first glance intact. Very glaring, however, was that K:\windows isn’t there. There is a new K:\found.000 folder with all the directory trees that lost their names (but otherwise fairly intact) re-rooted there. There were 30+ such re-rooted directories, and it was trivial to find the one that should be K:\windows.

Running through the directories on the disk image that had been combed over by chkdsk, everything was readable and accessible. I’m more worried about the data parts that got zero-filled by ddrescue, which makes for a much more insidious type of data corruption. On the other hand, I am comforted by the high percentage of data that must not be zero-filled based on the percentage of raw recovered bits recovered by ddrescue. Also, anything that chkdsk had a problem with (in the file table) must have been due to zero-filling. Thus, the more problems chkdsk found, the less problems remain in the data portion, so statistically I am satisfied. With NFI, the extent of the damage at the file level can be ascertained precisely, but for all practical purposes, the recovery is near total. Certainly, all the files that I care about (defined as those not “easily” regenerated), which was maybe 20GB of the 40GB, are recovered, including really small text files to fairly large (hundreds of MB) NTFS compressed files.

The ext2 physical structure consists of a basic unit called a block, whose size in bytes can be set at formatting time. Contiguous blocks are organized into groups, and contiguous groups make up the partition. The groups are of equal size, each containing by default 32768 blocks (on my partition a block is 4KB, so I guess a group is 128MB).

The next part is where I depart from the random wrong stuff strewn around the internet. I know they are wrong because I actually looked at the disk, but it caused me a lot of headache. The first group (Group 0) contains a 1KB filler, which some people like to call the “boot block” for unknown reasons (so far as I can tell, it doesn’t boot anything and it isn’t a block in size), followed by the 1KB Superblock (also not a block in size), which is like a filesystem header. It carries a bunch of file system parameters, and an identifying two-byte sequence called the “magic number.” At 4KB (for my block size) into Group 0 begins the Group Descriptor Table, which is also a filesystem header. The Group Descriptor Table contains 32-byte entries, one entry for each group on the disk, and each entry telling the location of the block bitmap (i.e. allocation table for blocks) and the inode bitmap (i.e. allocation table for inodes) belonging to that group.

The subsequent groups all have the same Group Descriptor Table at the same position in the group as in Group 0, and some of the groups also have a copy of the 1KB Superblock at its front (not at the 1KB position). The last part is important. The first two Superblocks are not 32768 blocks apart.

That is the gist of it. I didn’t bother with the block and inode allocation strategies because those aren’t important to me.

So when Knoppix said the Group Descriptor Table was corrupt, it means one or more of entries are pointing to the wrong places. Also when it says could not find the Superblock due to a “bad magic number,” it most likely means the Superblock copies in other groups are all overwritten, or, not where they are expected to be found, so they or the Group Descriptor Tables can’t be used to correct the Group 0 ones.

To find out what happened exactly, a disk editor was needed. I used to know a pretty good one for DOS, but I lost it. However, Windows Support Tools for Advanced Users has a Windows tool (used to be in the NT resource kit) called DiskProbe (dskprobe.exe) that does the trick. It’s a bit annoying that disk positions must be counted in units of sectors (512 B), but oh well. Also the Linux tool called fsstat was used, which dumps information read from the Superblock and Group Descriptor Table out one group at a time, whatever they contain (corrupt or not).

What I found, from fsstat, and from comparing with another disk formatted similarly with ext2, is all the Superblock copies in the groups that should have them are there, but offset by 1KB. How this happened I do not know. Seems like a bug of mkfs to me (it has to be mkfs because where the Superblock is supposed to be resides prior data). As for the source of the problem, the Group Descriptor Table stored in Group 0 has one entry in which 4 bytes were overwritten with 0x7FFFFFFF. How that happened I also do not know for sure, but it is almost certainly the fault of ext2ifs, since that is the only thing to touch the partition between it working and not working. But as 4 bytes appears to be the extent of the corruption, and the backup Group Descriptor Tables are perfectly good (they just can’t be found by silly Knoppix), I just copied four bytes by hand in DiskProbe, and bam, the ext2 file system was mountable again by Knoppix.

/media/sda2# ddrescue -B -n -i4000M /dev/hda rescue.image rescue.log

It continued to stick briefly at roughly 1GB intervals of data transfered, sometimes less. Sometimes I would manually interrupt the process and re-start it at a later point. This ddrescue allows with no problem because it uses its log intelligently, to patch together various chunks of recovered data, such that validly transferred data was never reread and interruptions were not costly to the recovered image. Wishing to avoid a repeat of the inextricable-3200MB-error from Sunday night, I interrupted ddrescue whenever it seemed to be stuck long. I noted the following sticking-points:

3277MiB
9798MiB
10415MiB
10917MiB
11906MiB
13885MiB
24499MiB
25890MiB
27394MiB
27851MiB
31280MiB
32799MiB
33748MiB
35065MiB
36954MiB

The data corruption pattern covers the entire disk — so far as there is any physical correspondence — but affects only a small percentage of disk’s data. In fact, the recovery to error ratio was at least 100:1. I eventually transferred about 38GB of raw bits out of the 40GB disk.

By the way, the -n option demands “no error splitting” meaning that a 64K-chunk of data with any read problems is marked as wholly erroneous (or “/” in the log file… good data is “+” in the log file). After the runs with -n, I did a run with error splitting on a small portion on the disk, forcing ddrescue to analyze only the 64K-chunks with errors on the original run. With this method, I recovered yet more data (“/” became mostly “+” and some “-”).

TRYING THE RAW IMAGE UNDER WINDOWS

Before attacking more chunks with error-splitting, I decided to examine the current state of the disk image. There would be no point to try very hard on the bad parts if the useful files were already contained in the “+” regions. Just from the log file, I was pleased to find the first 5K intact (this includes the first sector holding the MBR and partition table), then many errors in the front of the drive which were in the Dell Diagnostic partition, followed by several hundred MB intact, which would include the boot sector of the NTFS partition and the front of its master file table hopefully. A chunk of the end of the drive was also intact, which should contain the backup NTFS boot sector. Things were looking much better.

So I next planned to mount the image under Windows, since many NTFS analysis tools run only on Windows. To do this, I needed a filesystem driver to read ext2 under Windows. There are a few, but the “ext2 Installable File System for Windows” or ext2ifs boasts Kernel-mode extension of the Windows file system that “is indeed comparable to Windows NT’s native file system drivers“. Not really, as the implementation is missing some behavior one would expect from Windows but it does behave in such a way that most of the usual higher-level filesystem manipulations can be done directly on the ext2 volume, including getting real drive letters and the ability to perform file manipulation directly (without requiring a specialized file manager).

First I wanted to make a copy of the image file, for two reasons: to have a copy to work with without endangering the recovered data, and to put the original back into ddrescue to churn away at erroneous chunks according to which chunks turn out to be important. In retrospect I could have made this copy under Linux, but since I had ext2 already mounted in Windows, I just made the copy under Windows. While it was doing this copying, guess what, the source drive (the external USB drive) became unreadable. Uh……

I took the drive with the 38GB of good data back into Knoppix Linux, and Linux could not mount it either. Uh……

# dmesg | tail

“Corrupt group descriptor: bad block for inode bitmap” etc. etc. Uh……

What did the ext2ifs driver do? There is not a single reason it should have touched the group descriptor table.

# e2fsck /dev/sda2

“e2fsck: Bad magic number in super-block while trying to open /dev/sda2
The superblock could not be read or does not describe a correct ext2 filesystem.” etc. etc. Uh……

What did mkfs.ext2 do? There are backups of these file system parameters, why are they all bad now?

What did the USB enclosure do? It didn’t barf all over the disk, did it?

Many a self-proclaimed miracle recovery boot disks/discs conceivably take care of both tasks. A quick search turns up a boot floppy software called “EaseUs disk copy” using a self-booting “freedos” but it was useless because it would like to copy within a computer (from master IDE drive to slave). Into the trash it goes. There were a few others like this, all useless.

Next up, I’ve got some Windows-based disk editor tool. Maybe I can install Windows on this external USB hard disk so I can boot the laptop? No. Evidently it isn’t supported. Windows installed fine on the external USB drive (all files are copied), but on the first boot BSOD’d. Some say there is a way to hack the Windows CD to make this work. Well I don’t have that much time to waste, so forget that.

Next up, Knoppix version 5 (Linux Knoppix 2.6.17 #4 SMP PREEMPT, i686 GNU/Linux). I had some difficulty mounting the external USB hard drive, but finally by booting from CD, then attaching the external drive after startup, Knoppix found it and stuck it under /dev/sdaN (for N in 1,2,etc.). The drive had a 20GB FAT32 partition already (where I attempted to install Windows before), along with an empty 120GB partition, so the empty partition was at /dev/sda2.

These days, the favorite recovery tool under Knoppix seems to be the ignore-on-read-error disk copy tool called ddrescue (not the Knoppix-included dd_rescue). I believe the difference is that ddrescue is much more automatic in dealing with the log file — this is important. Both will copy a disk’s contents into a disk image file. ddrescue needs to be compiled though; fortunately compilation was clean.

The choice of filesystem for /dev/sda2, which would hold the disk image, was limited. The plan was to eventually read the data on a Windows computer, which would be easiest with FAT32 or NTFS. However, NTFS write-support on Knoppix is of unknown reliability (– why? I don’t know). I tried FAT32, using mkdosfs to format because Windows refuses to format a FAT32 volume larger than 32GB. Then I even began copying data before realizing, to my chagrin, that FAT32 has another limit: 4GB on the file size. That left ext2, and its default block-size of 4K was fine (block-size under 2K would be a problem due to file size limitation), so something like:

# mkfs.ext2 /dev/sda2
# mount -t ext2 /dev/sda2 /media/sda2

Progress.

Now, computer off. Broken Seagate goes back in. Boot. And… no boot. The boot sequence now stuck at two points where it wanted to read the Seagate drive (on /dev/hda): First, it attempted to find a (nonexistent) CD on /dev/hda, resulting in churning the hard drive for a few seconds (damaging my data further, I’m sure!) Next, at the auto-detection stage to create /etc/fstab, Knoppix repeatedly tried to read the drive, and apparently would go no further. Online sources are rife with suggestions of passing the boot options “nofstab” and “noswap” to the kernel to solve this problem. Wrong. With these options set, it still had the same problem at the auto-creation of /etc/fstab. The solution turned out to be nail-biting patience: After booting and allowing the auto-detection to proceed for 10+ minutes of now-familiar churn-churn-churn on the broken Seagate, heating it up and losing me more data, it eventually did go on and boot. Um… whoever is responsible for this boot code should be flogged or at least, made to go home and work on it some more. I can think of zero (0) reasons why this particular process warrants persisting through multiple disk read errors.

Now, /dev/hda wouldn’t mount (of course), but no matter. Here we have:

/media/sda2# ddrescue -B -n /dev/hda rescue.image rescue.log

Next thing to do was to boot off the Windows XP CD, hoping to use its Recovery Console to see if the file system is still okay, and maybe try to fix the file system if the problem isn’t too serious. At this point however, my CD drive made a loud clicking noise, and I found the spindle plastic at the axis of the CD had come off the drive. Two malfunctions on the same day? Woe is me. What’s wrong with this picture — I don’t believe I killed any small animals today (or on any day really)!

So I pushed the spindle plastic back on, tried again. No luck. I conclude that the CD-ROM, my only other means of mass-input, is broken just when I need it quite much. In my last attempt to fix anything today, I tried white-out as glue on the plastic part. This worked. Unfortunately, the CD drive vibrates far more than before, due to its spindle being now slightly off-kilter. This isn’t going to last for very long…