Windows 7 Math Input Panel!
Comments(1)
Somehow the ability to turn handwritten math into MathML escaped my attention as a Windows 7 feature from trying the first beta. Finally! I’ve been waiting for this since forever… Wonder what took so long.
Next up are music notation and graphics in general*. The ultimate goal of a handwriting recognizer is of course similar to that of OCR: to turn one piece of art (for the lack of a better word) rendering to another, text included. Specifically, it should rectify all the rendering to a “typeset” form. It should intelligently recognize a host of objects with its own Visio-like templates: if I draw a resistor, it should pull out a nice schematic rendering of a resistor. If I draw a rectangle, and select “rectify”, it should make a rectangle with straight edges.
It is in some sense harder than OCR: the handwriting can vary a lot; but in other sense easier: the user is by definition actively interacting with the process. Combined with a rational input device (not a mouse, not a pen that is but a mouse emulator**), this can become the realistic next-generation human-computer interface, and tablet will take off as one of the next-generation form factors (as it almost inevitably must take off, probably in a merged tablet/ebook form). I’m not too sure what human-computer interface people are doing these days, but last I checked they were working on fanciful stuff divorced from an actual use. If it were up to me to decide, I’d explore the limits of the pen input device first, simple things often yield greater and surprising results.
* Now this is something I’ve been ranting about for years as one of these “obvious” advances that should have happened but haven’t, but it is also possible that technology has caught up enough to implement these things. Certainly the math input idea isn’t new at all. Somebody wrote an MEng thesis on it 10 years ago and I’m sure he wasn’t the first one. The general input problem is more interesting though.
** A useful pen device would forsake the god-awful mouse model, and move along the lines of some pens nowadays: with a digital eraser, with pressure sensing (these two already exist), but also with a scroll wheel to select menus or choose options, with some feedback either mechanical or like a small display. The latter can be useful, for instance, to show the color of the pen or some other such property. There is no reason why a pen cannot be as useful — indeed, more useful — than a mouse (which has become an uncomfortable and unnatural inner glove). The pen form factor has been tried for thousands of years and its usefulness should not be doubted, disdain for archaism be damned.
[...] So let me make it very concrete. I take out a piece of hardware — it is a tablet. I would like to write a paper. But I have coauthors. So I start a new document — maybe it’s for writing LaTeX. I start writing — with a pen — or I type, whatever. At some point I decide I want collaboration, so maybe I turn this into a “collaboration” mode. Think of this as going to a public space to write on the whiteboard. Now anybody can see this in progress, in real time, not as its autosaved version, if open up the same program of theirs and I’m on their collaborator list or something. Maybe I tell them out of band that I want to collaborate or they already know. They can request to join, and I can let them, and even add them to the list of collaborators for this document (or project, as every document should be automatically versioned into a project tree). They can edit at the same time as I edit, and each person can see what the other person is doing, if they just went to the right part of the document to look. The part you are editing can be locked if you wish. Or you can lock other parts, doesn’t matter. You can make comments on the margins. You can run a view of the current snapshot any time, without interfering with other editors. At any time, you can save a state as a satisfactory “version”. References can be added by dropping anything — PDF, URL, some search text — and the editor can go look for the reference and turn it into the right format, and cache it into your library of references. If you want to draw figures, you should be able to do it in place, with a pen, and it will be turned into nice figures (discussed separately before)… etc. etc. [...]