Somehow the ability to turn handwritten math into MathML escaped my attention as a Windows 7 feature from trying the first beta. Finally! I’ve been waiting for this since forever… Wonder what took so long.

Next up are music notation and graphics in general*. The ultimate goal of a handwriting recognizer is of course similar to that of OCR: to turn one piece of art (for the lack of a better word) rendering to another, text included. Specifically, it should rectify all the rendering to a “typeset” form. It should intelligently recognize a host of objects with its own Visio-like templates: if I draw a resistor, it should pull out a nice schematic rendering of a resistor. If I draw a rectangle, and select “rectify”, it should make a rectangle with straight edges.

It is in some sense harder than OCR: the handwriting can vary a lot; but in other sense easier: the user is by definition actively interacting with the process. Combined with a rational input device (not a mouse, not a pen that is but a mouse emulator**), this can become the realistic next-generation human-computer interface, and tablet will take off as one of the next-generation form factors (as it almost inevitably must take off, probably in a merged tablet/ebook form). I’m not too sure what human-computer interface people are doing these days, but last I checked they were working on fanciful stuff divorced from an actual use. If it were up to me to decide, I’d explore the limits of the pen input device first, simple things often yield greater and surprising results.

* Now this is something I’ve been ranting about for years as one of these “obvious” advances that should have happened but haven’t, but it is also possible that technology has caught up enough to implement these things. Certainly the math input idea isn’t new at all. Somebody wrote an MEng thesis on it 10 years ago and I’m sure he wasn’t the first one. The general input problem is more interesting though.

** A useful pen device would forsake the god-awful mouse model, and move along the lines of some pens nowadays: with a digital eraser, with pressure sensing (these two already exist), but also with a scroll wheel to select menus or choose options, with some feedback either mechanical or like a small display. The latter can be useful, for instance, to show the color of the pen or some other such property. There is no reason why a pen cannot be as useful — indeed, more useful — than a mouse (which has become an uncomfortable and unnatural inner glove). The pen form factor has been tried for thousands of years and its usefulness should not be doubted, disdain for archaism be damned.