The other task was more interesting from a learning perspective. When a file is to be encrypted by NTFS, it generates a file-specific symmetric “file encryption key” and stores the key with the file. Of course the key isn’t stored in plaintext but is encrypted using the user’s public key and is only decodeable by the user’s private key. Normally the user should export the private key into a .pfx certificate file and store it on a smart card or USB pen drive or something, and more importantly, remove it from the certificate store on Windows. I did make a copy of the private key off the computer. I could just re-import the certificate, and then decrypting the data would be trivial. But the external storage device that holds the private key isn’t with me during the break, unfortunately. On the other hand, I didn’t bother to remove the private key from the computer either, because I wasn’t really serious about this particular piece of data, just toying around with the NTFS encryption feature, mostly.

That made the encrypted file potentially recoverable without delay. Windows doesn’t store the user’s private key in plaintext either, of course. From what I understand, the user’s private key is encrypted somehow with user’s password hash, then maybe it is encrypted again with a master key belonging to the system. Somehow, the system’s master key is the weak link. I think it is stored in the SAM database of the system registry, which is in %SYSTEMROOT%\system32\config, though I may be mistaken on this point. The user’s various keys are stored in %USERPROFILE%\Application Data\Microsoft\Protect and %USERPROFILE%\Application Data\Microsoft\Crypto. Here is another explanation of this.

To make a long story short, if all of these files are available, and the user’s password is known, then it is possible to recover the encrypted files directly from the drive without an explicitly exported private key (hence encryption with these files around is ultimately pointless).

I ended up using the Elcomsoft Advanced EFS Data Recovery tool (AEFSDR). They have a trial version, too. Unfortunately it’s crippleware, unlike Mount Image Pro. All I can say is, if you really want to decrypt files, you find a way around the “cripple” in crippleware. Enough said.

For the record, AEFSDR is pretty good and fast. However, I am also not completely satisfied by it. I haven’t found any bugs, but the problem with it is its incompleteness. I don’t know what file system access method it uses, but it cannot see any removable drives, including removable USB drives or any mounted virtual disks (e.g. by Mount Image Pro). This is severely limiting, because encrypted files are not easily copied (Windows says Permission Denied), and I basically ran into a wall because of this. Please, AEFSDR, go work on this some more. You are selling a recovery tool, so your users are most likely in desperate straits with limited resources… they don’t want to deal with your programming kludges.

The way around it is to use NTBACKUP (part of Windows), which can archive encrypted files to a backup image, and then you can restore these files onto a permanent physical disk. After also copying the security related files from Windows to a permanent physical disk, AEFSDR decrypts all files successfully.

RELOADING THE DDRESCUE IMAGE

After I got as much as I am ever going to get out of the Seagate disk using ddrescue (I got another few tens of megabytes out of the disk), I retraced the steps on the new image. This time, the image loads as a valid NTFS partition and Windows will read it (apparently the last few bits recovered was pretty important). However, the Windows directory is still not readable, although almost all other directories are directly readable. NFI also works on the image — this is very good, as it means I can finally profile what I lost down to the last sector. I also ran chkdsk on it again — fewer errors to correct this time, but hopefully fewer zero-filled data issues to deal with at the end of the day. This chkdsk’d image is now the gold recovery image. I backed it up immediately.

Then I took the copy of the image under Windows. This way I didn’t care what the ext2ifs driver wanted to do. But this time it didn’t mangle the partition (probably because this ext2 partition actually has a legal physical formatting!).

Finally I can run NTFS tools on the broken disk, but I need to mount the (broken) NTFS image first. There is a free tool, FileDisk by Bo Branten, to do this. Unfortunately, it gives a drive letter to the whole disk, instead of to the partitions. This is nearly impossible to work with because (1) there are 63 sectors of MBR, partition table, and filler at the front of the disk, and (2) there is the Dell diagnostic partition. And FileDisk doesn’t virtualize a physical disk device, which would be the more correct metaphor.

A way around this is to use VMWare, and virtualize the disk image as a disk. But no, that’s too much trouble… although, after this incident, I’m really considering putting a VMWare image with Windows preinstalled onto a DVD or something — that would be the complement to Knoppix.

A somewhat more advanced tool called Mount Image Pro (MIP) exists that does what I want. It isn’t free (*), but who cares when there is a one-month trial? (For the record, I’m not entirely happy with this thing, either. It works, but almost every other time it fails to load the system driver and requires a reboot and retry.)

With MIP, the NTFS partition comes up and is mounted under K:, but it is completely broken as expected. The file system cannot be read normally by Windows, nor would the diagnostic tool NFI (part of the OEM Support Tools) work on it. NFI relates what files a sector contains and what sectors a file resides in.

The next step is then to run chkdsk:

> chkdsk /v K:

Very specific errors relating to the file table are immediately detected and chkdsk says it cannot continue in the (default) read-only mode.

So let’s try

> chkdsk /v /r K:

Same complaint by chkdsk. What? It turns out MIP (at least the GUI) only mounts images in read-only mode, but it doesn’t specify this anywhere! Nor does it give an indication of how to get around this. Wow great.

After poking around, I noticed there is a command-line interface to MIP, too, and it is there that you can specify mounting in “read-write” mode, instead of “write-block” mode. (I would also call “write-block” mode “read-only” mode, but that’s just me, so what do I know!) Moving on:

> mip mount rescue.image /rw /p:2 /l:K

Finally, the second partition in the disk image is mounted on the drive letter K: (hence the last two parameters). Unfortunately, MIP screws up again and says there is only “1″ partition, and the drive letter K is associated with the first, Dell diagnostic, partition… even though it clearly displays two partitions and that the drive letter K is in reality associated with the second partition. No matter, it still works, so I’ll leave the MIP people unflogged. But they should still go home and fix these.

Trying chkdsk again and it slaves away through a 5-step process, dumping output on all the files and directories on which it detected problems. The disk image is broken enough that chkdsk had to kill most of the NTFS file permission. After about 2 or 3 hours, K: is miraculously readable again in Windows, with the basic directory structure at first glance intact. Very glaring, however, was that K:\windows isn’t there. There is a new K:\found.000 folder with all the directory trees that lost their names (but otherwise fairly intact) re-rooted there. There were 30+ such re-rooted directories, and it was trivial to find the one that should be K:\windows.

Running through the directories on the disk image that had been combed over by chkdsk, everything was readable and accessible. I’m more worried about the data parts that got zero-filled by ddrescue, which makes for a much more insidious type of data corruption. On the other hand, I am comforted by the high percentage of data that must not be zero-filled based on the percentage of raw recovered bits recovered by ddrescue. Also, anything that chkdsk had a problem with (in the file table) must have been due to zero-filling. Thus, the more problems chkdsk found, the less problems remain in the data portion, so statistically I am satisfied. With NFI, the extent of the damage at the file level can be ascertained precisely, but for all practical purposes, the recovery is near total. Certainly, all the files that I care about (defined as those not “easily” regenerated), which was maybe 20GB of the 40GB, are recovered, including really small text files to fairly large (hundreds of MB) NTFS compressed files.