The other task was more interesting from a learning perspective. When a file is to be encrypted by NTFS, it generates a file-specific symmetric “file encryption key” and stores the key with the file. Of course the key isn’t stored in plaintext but is encrypted using the user’s public key and is only decodeable by the user’s private key. Normally the user should export the private key into a .pfx certificate file and store it on a smart card or USB pen drive or something, and more importantly, remove it from the certificate store on Windows. I did make a copy of the private key off the computer. I could just re-import the certificate, and then decrypting the data would be trivial. But the external storage device that holds the private key isn’t with me during the break, unfortunately. On the other hand, I didn’t bother to remove the private key from the computer either, because I wasn’t really serious about this particular piece of data, just toying around with the NTFS encryption feature, mostly.

That made the encrypted file potentially recoverable without delay. Windows doesn’t store the user’s private key in plaintext either, of course. From what I understand, the user’s private key is encrypted somehow with user’s password hash, then maybe it is encrypted again with a master key belonging to the system. Somehow, the system’s master key is the weak link. I think it is stored in the SAM database of the system registry, which is in %SYSTEMROOT%\system32\config, though I may be mistaken on this point. The user’s various keys are stored in %USERPROFILE%\Application Data\Microsoft\Protect and %USERPROFILE%\Application Data\Microsoft\Crypto. Here is another explanation of this.

To make a long story short, if all of these files are available, and the user’s password is known, then it is possible to recover the encrypted files directly from the drive without an explicitly exported private key (hence encryption with these files around is ultimately pointless).

I ended up using the Elcomsoft Advanced EFS Data Recovery tool (AEFSDR). They have a trial version, too. Unfortunately it’s crippleware, unlike Mount Image Pro. All I can say is, if you really want to decrypt files, you find a way around the “cripple” in crippleware. Enough said.

For the record, AEFSDR is pretty good and fast. However, I am also not completely satisfied by it. I haven’t found any bugs, but the problem with it is its incompleteness. I don’t know what file system access method it uses, but it cannot see any removable drives, including removable USB drives or any mounted virtual disks (e.g. by Mount Image Pro). This is severely limiting, because encrypted files are not easily copied (Windows says Permission Denied), and I basically ran into a wall because of this. Please, AEFSDR, go work on this some more. You are selling a recovery tool, so your users are most likely in desperate straits with limited resources… they don’t want to deal with your programming kludges.

The way around it is to use NTBACKUP (part of Windows), which can archive encrypted files to a backup image, and then you can restore these files onto a permanent physical disk. After also copying the security related files from Windows to a permanent physical disk, AEFSDR decrypts all files successfully.

RELOADING THE DDRESCUE IMAGE

After I got as much as I am ever going to get out of the Seagate disk using ddrescue (I got another few tens of megabytes out of the disk), I retraced the steps on the new image. This time, the image loads as a valid NTFS partition and Windows will read it (apparently the last few bits recovered was pretty important). However, the Windows directory is still not readable, although almost all other directories are directly readable. NFI also works on the image — this is very good, as it means I can finally profile what I lost down to the last sector. I also ran chkdsk on it again — fewer errors to correct this time, but hopefully fewer zero-filled data issues to deal with at the end of the day. This chkdsk’d image is now the gold recovery image. I backed it up immediately.

The ext2 physical structure consists of a basic unit called a block, whose size in bytes can be set at formatting time. Contiguous blocks are organized into groups, and contiguous groups make up the partition. The groups are of equal size, each containing by default 32768 blocks (on my partition a block is 4KB, so I guess a group is 128MB).

The next part is where I depart from the random wrong stuff strewn around the internet. I know they are wrong because I actually looked at the disk, but it caused me a lot of headache. The first group (Group 0) contains a 1KB filler, which some people like to call the “boot block” for unknown reasons (so far as I can tell, it doesn’t boot anything and it isn’t a block in size), followed by the 1KB Superblock (also not a block in size), which is like a filesystem header. It carries a bunch of file system parameters, and an identifying two-byte sequence called the “magic number.” At 4KB (for my block size) into Group 0 begins the Group Descriptor Table, which is also a filesystem header. The Group Descriptor Table contains 32-byte entries, one entry for each group on the disk, and each entry telling the location of the block bitmap (i.e. allocation table for blocks) and the inode bitmap (i.e. allocation table for inodes) belonging to that group.

The subsequent groups all have the same Group Descriptor Table at the same position in the group as in Group 0, and some of the groups also have a copy of the 1KB Superblock at its front (not at the 1KB position). The last part is important. The first two Superblocks are not 32768 blocks apart.

That is the gist of it. I didn’t bother with the block and inode allocation strategies because those aren’t important to me.

So when Knoppix said the Group Descriptor Table was corrupt, it means one or more of entries are pointing to the wrong places. Also when it says could not find the Superblock due to a “bad magic number,” it most likely means the Superblock copies in other groups are all overwritten, or, not where they are expected to be found, so they or the Group Descriptor Tables can’t be used to correct the Group 0 ones.

To find out what happened exactly, a disk editor was needed. I used to know a pretty good one for DOS, but I lost it. However, Windows Support Tools for Advanced Users has a Windows tool (used to be in the NT resource kit) called DiskProbe (dskprobe.exe) that does the trick. It’s a bit annoying that disk positions must be counted in units of sectors (512 B), but oh well. Also the Linux tool called fsstat was used, which dumps information read from the Superblock and Group Descriptor Table out one group at a time, whatever they contain (corrupt or not).

What I found, from fsstat, and from comparing with another disk formatted similarly with ext2, is all the Superblock copies in the groups that should have them are there, but offset by 1KB. How this happened I do not know. Seems like a bug of mkfs to me (it has to be mkfs because where the Superblock is supposed to be resides prior data). As for the source of the problem, the Group Descriptor Table stored in Group 0 has one entry in which 4 bytes were overwritten with 0x7FFFFFFF. How that happened I also do not know for sure, but it is almost certainly the fault of ext2ifs, since that is the only thing to touch the partition between it working and not working. But as 4 bytes appears to be the extent of the corruption, and the backup Group Descriptor Tables are perfectly good (they just can’t be found by silly Knoppix), I just copied four bytes by hand in DiskProbe, and bam, the ext2 file system was mountable again by Knoppix.