The history of synthesized instruments has ping-ponged between the two methods. First there was FM synthesis, which used analog function generation based on the simplest physical models of standing waves (harmonics, etc.). This allowed distinguishing between major instrument groups but sounded decidedly fake. Then people recorded acoustic instruments and looped/interpolated between samples — much better, but storage constraints placed limits on what could be processed; and there was never any dynamism. Then it was back to physical modeling, this time using waveguides to determine how various inputs like bowing on strings or blowing into pipes dynamically affect output sound (I think it started at CCRMA). This gave really good expressivity — but again sounded fake. And so back to wave-samples. For the last 15 years or so, especially with the cheapening of storage, it appears that the dumbest, brute-force method of using a large enough number of samples and ad-hoc adjustments to their decay and reverb characteristics became dominant. For ensemble instruments with little individual personality, it was actually superb. The trouble was always with instruments in solo.

One has to ask the question though — is sampling really that dumb?

In other words, is physical modeling ever going to achieve reality? Sure, with more processing power, more realistic models could be built. But let’s be serious: the reason why acoustic instruments sound real is likely because they are built exactly the way they are. They are already made (physically) as simple as they can be — e.g. just some metal pipes or a piece of string, so that a “realistic” model may well turn out to be the acoustic object itself, with all of its mechanical physics…

Instead of trying to reverse engineer the box that takes mechanical input into sound output, we may as well embrace sampling by taking a huge number of input and output pairs. But we should do it intelligently. The entire space shouldn’t be sampled evenly or by hunches. It should be more like what they do at CCRMA where they attempt to sample the various independent modes of an instrument — striking it with a mallet at different locations to get impulse responses, for example.

Beyond this, we should consider decomposing sound reproduction differently. The sound generation process need not be fully decoupled at the point of digital waves, that then need to be reproduced by generic membrane-based speakers. Perhaps we should instead take acoustic objects that span various modes of the output space — things like a large piece of wood, a small piece of wood, some metal pipe, some pins, whatever minimal set we could derive (may even be a weird basis set of synthetically shaped physical objects) — then drive them like specialized “speakers.” Likewise, we do the same modal decomposition with input interfaces — a pressure-resistive mouthpiece, a coupled string, etc.

Much like in color science, there should be the concept of gamut that a particular output device can cover and that a particular input device can pick up. Then we should match real sounds from real instruments with the inputs/outputs we get on our input/output devices by applying the appropriate digital transforms, which we get by (mathematically) solving the inverse problem from measurements, rather than trying to come up with the physics from first principles.

What we get is then a universal instrument that should reproduce acoustic instruments (and many other realizable timbres) accurately. As a bonus, it has the benefit of allowing interchangeable “playing” styles upon swapping inputs (cf. keytar, melodica).

Tensors are sums of “products” of vectors. There are different kinds of vector products. The one used to build tensors is, naturally, the tensor product. In the Cartesian product of vector spaces \(V\times W\), the set elements are tuples like \((v,w)\) where \(v\in V, w\in W\). A tensor product \(v\otimes w\) is obtained by tupling the component bases rather than the component elements. If \(V\) has basis \(\{e_i\}_{i\in\{1,…,M\}}\) and \(W\) has basis \(\{f_j\}_{j\in\{1,…,N\}}\), then take \(\{(e_i,f_j)\}_{i\in\{1,…,M\},j\in\{1,…,N\}}\) as the basis of the tensor product space \(V\otimes W\). Then define the tensor product \(v\otimes w\) as

(1) \(\sum_{i,j} v_i w_j (e_i,f_j) \in V\otimes W\),

if \(v=\sum_i v_i e_i\) and \(w=\sum_j w_j f_j\). The entire tensor product space \(V\otimes W\) is defined as sums of these tensor products

(2) \(\{\sum_k v_k\otimes w_k | v_k\in V, w_k\in W\}\).

So tensors in a given basis can be represented as multidimensional arrays.

\(V\otimes W\) is also a vector space, with \(MN\) basis dimensions (c.f. \(V\times W\) with \(M+N\) basis dimensions). But additionally, it has internal multilinear structure due to the fact that it is made of component vector spaces, namely:

\((v_1+v_2)\otimes w = v_1\otimes w + v_2\otimes w\)
\(v\otimes (w_1+w_2) = v\otimes w_1 + v\otimes w_2\)
\(\alpha (v\otimes w) = (\alpha v)\otimes w = v\otimes (\alpha w)\)

Higher-order (n-th order) tensor products \(v_1\otimes v_2\otimes \cdots \otimes v_n\) are obtained by chaining in the obvious way, likewise for higher-order tensor product spaces \(V_1\otimes V_2\otimes \cdots \otimes V_n\). With this, concatenation of tensors are also defined, i.e. \(S_{i_1,…i_m} \in V_1\otimes \cdots \otimes V_m\) and \(T_{i_{m+1},…,i_n} \in V_{m+1}\otimes \cdots \otimes V_n\), then \(S_{i_1,…,i_m}\otimes T_{i_{m+1},…,i_n} = Z_{i_1,…,i_n} \in V_1\otimes \cdots \otimes V_n\). In other words, the indices are appended. This is essentially the Kronecker product, which generalizes the outer product.

However, usually when tensors are mentioned, the tensor product spaces under discussion are already specialized to those generated from a single base vector space \(V\) and its dual space \(V^*\), rather than from a collection of arbitrary vector spaces. In such a space \(P(m,n) = \overbrace{V\otimes \cdots \otimes V}^{m} \otimes \overbrace{V^*\otimes \cdots \otimes V^*}^{n}\), the component spaces (and their bases, indices, etc.) naturally belong to two groups, those from \(V\) are called contravariant, those from \(V^*\) are called covariant, and an (m,n)-tensor from \(P(m,n)\) is written \(T^{i_1,…,i_m}_{j_1,…,j_n}\).

This specialization allows the contraction of tensors to be defined. A contraction basically chooses one covariant vector component and one contravariant vector component from a tensor and applies the former as a functional on the latter, e.g., contracting \(T^{i_1,…,i_m}_{j_1,…,j_n} = v_{i_1}\otimes \cdots \otimes v_{i_m} \otimes v^*_{j_1} \otimes \cdots \otimes v^*_{j_n}\) on the pair of indices \(i_1\) and \(j_1\) gives \(Z^{i_2,…,i_m}_{j_2,…,j_n} = v^*_{j_1}(v_{i_1}) (v_{i_2}\otimes \cdots \otimes v_{i_m} \otimes v^*_{j_2} \otimes \cdots \otimes v^*_{j_n})\). \(v^*_{j_1}(v_{i_1})\) of course is an inner product that sums over the dimensions of the paired components. Contraction generalizes the trace operator. Combined with concatenation, this defines a tensor multiplication, such that if \(S^{r,i_2,…,i_m}_{s,j_2,…,j_n}\in P(m,n)\) and \(T^{s,k_2,…,k_p}_{r,l_2,…,l_q}\in P(p,q)\), then \(S^{r,i_2,…,i_m}_{s,j_2,…,j_n}T^{s,k_2,…,k_p}_{r,l_2,…,l_q}\) is the contraction of \(S^{r,i_2,…,i_m}_{s,j_2,…,j_n}\otimes T^{s,k_2,…,k_p}_{r,l_2,…,l_q}\) on all common indices that can be paired, e.g. \(r,s\). This is the so-called Einstein notation, and generalizes matrix multiplication.

The distinction of \(V\) vs. \(V^*\) also manifests in the change-of-basis rules for tensors, which inherit from the change-of-basis rules of the component vector spaces, which are:

• contravariant change-of-basis rule: If \(B = [b_1\ b_2\ \cdots\ b_M]\) is a change-of-basis matrix, with the new basis \(\{b_i\}_{i\in \{1,…,M\}}\) written in the old basis as columns, then for a vector written in the old basis \(v\in V\) and the same vector written in the new basis \(\tilde{v}\in V\), \(v = B\tilde{v}\). Therefore, we have

  (3) \(v \mapsto \tilde{v} = B^{-1}v\).

• covariant change-of-basis rule: If additionally \(a^T\in V^*\) is a functional written in the old basis and \(\tilde{a}^T\in V^*\) is the same functional written in the new basis, then \(\forall v\in V: a^T v = \tilde{a}^T \tilde{v} = \tilde{a}^T B^{-1}v\). Therefore, we have

  (4) \(a^T \mapsto \tilde{a}^T = a^T B\).

Combining (1), it can be shown that, for a change-of-basis tensor \(B = {B^{-1}}^{i_1}_{i_1}\cdots {B^{-1}}^{i_m}_{i_m}B^{j_1}_{j_1}\cdots B^{j_n}_{j_n}\), an (m,n)-tensor \(T^{i_1,…,i_m}_{j_1,…,j_n}\) has the change-of-basis rule \(T^{i_1,…,i_m}_{j_1,…,j_n} \mapsto BT^{i_1,…,i_m}_{j_1,…,j_n}\).

Okay, so what’s the point of these tensors? Basically, an (m,n)-tensor \(T^{i_1,…,i_m}_{j_1,…,j_n}\in P(m,n)\) represents a multilinear input-output relationship that takes \(n\) vectors as input and produces \(m\) vectors as output. If used “canonically” on an input \(X^{j_1,…,j_n}\in P(n,0)\), you get \(T^{i_1,…,i_m}_{j_1,…,j_n}X^{j_1,…,j_n} = Y^{i_1,…,i_m}\in P(m,0)\) as output. The contravariant input gets contracted with the covariant parts of the transformation tensor, and these drive the contravariant parts of the transformation tensor to produce the contravariant output.

(System diagrams: Rank is the minimal number of terms in a tensor. On the left, a rank-1 tensor transformation; on the right, a rank-\(K\) one. )

An example is linear transformations, which are (1,1)-tensors (1 vector in, 1 vector out). In array representation these would just be matrices. Any rank-\(K\) linear transformation \(T^v_a\) is decomposable into a \(K\)-term tensor \(\sum_{k=1}^K v_k\otimes a_k\), but 1-term (1,1)-tensors are outer products, so this is the matrix \(\sum_{k=1}^K v_k a^T_k\), and \(Y^v = T^v_a X^a\) is just \(y = \sum_k v_k (a^T_k x)\).

Most other “multilinear” operations on vectors (inner product, cross product, wedge product, determinant) can be written as tensors. For example, the inner product operation (2 vectors in, “0″ vectors out, i.e. scalar) is the (0,2) \(N\)-term Kronecker tensor \(\delta_{i_1 i_2}=\sum_{k=1}^N e^*_k\otimes e^*_k\) where \(\{e^*_k\}_{k\in\{1,…,N\}}\) are the standard basis of \(V^*\).

Eduardo … I’m talking about taking the entire social experience of college and putting it online.

When the movie “The Social Network” came out, this line caught my attention. I’m not sure this thesis — let’s call it the “replication thesis” — was what Monsieur Zuckerberg had in mind rather than something the screenwriter came up with, but it makes sense as to what actually undergirds online social platforms of today.

In all likelihood, Zuckerberg did not at first intend Facebook to be more than its namesake — a dorm facebook. Just as, in all likelihood, Twitter was meant as no more than a status message broadcast system, at first. The fact that Facebook became something of a gathering place and Twitter became a “microblogging” service — in essence, taking over functions that used to be conducted in other ways — I think owed something to their use of a “correct” user graph for certain contexts. It was the user graph that allowed, then limited, the range of social functions that people were willing to port over to the online platform. With the undirected graph, Facebook (and clones) modeled something like a big gathering, maybe a party. With the directed graph, Twitter (and clones) modeled something a bit more nuanced, like a groupie-celebrity relationship. (Is it any surprise, then, that celebrities drove the latter’s popularity?)

But I get the sense that neither Facebook nor Twitter truly believes in the replication thesis. They’ve construed their challenge narrowly as one of periodically pushing out new “things you could do,” most of which are nowadays ignored by users, or adding more places at which you could interact, but in the same way. They don’t see that users voluntarily do on a platform only those things that are compatible with their perception of the modeled social space. You can’t push anything on them any more than you can force people to play some game at a party. Yet I see no movement to revisit the user graph and better model real social relationships with all of their complexities. If left unchanged, the inevitable result will be that the range of social functions these platforms support stagnates, and therein should lie their eventual downfall. In fact, that probably solves the supposed “mystery” of Myspace’s decline, too. It is in this context that Google+ arrives.

Why has Google+ “succeeded” in a way that Orkut and Buzz had not? For one, Google+ posits an even more complex user graph in the form of circles. It’s a colored directed graph, not only directed like Twitter, but each user’s edges to other users are colored by the circles these neighbors belong to. Instantly, the range of social functions is expanded. We don’t have to wait for new applications to take advantage of this to make the realization: imagined possibilities are sufficient. It is like going to a new space. You leave the party, and the groupie session, and come to this space that is novel in a way that is defined not by its location or the people in it, but by its different arrangement.

By accident, Google is also a provider of several useful web services. This is an enviable position from which to make suggestive proposals on new interactions that people can have on this particular user graph. And unlike on Facebook or Twitter, some of Google’s more productive collaborative applications may actually require the Google+ user graph. The former two platforms are already shackled by their user graphs to “unserious” applications that are not coincidentally popular there. However, while Google+ will have a while to go before it exhausts these unrealized possibilities, it isn’t immune to the same questions of what is a “correct” user graph model or, what “correctly” replicates the “entire social experience”. Colored directed graph models social relationships in greater detail but is this model complete, or fundamentally right?

I recall reading Zuckerberg’s recent comment in which he quipped (about Google+) that it would be too much trouble to keep track of many circles, and that Facebook Groups is more transparent in that everybody knows who else is in the group (again, like at a party). I think he’s on to something even if his own model isn’t really moving forward. The fact that you can put people into circles gives you a false sense of privacy, because there will always be common connections and information leak. For example, you can go to a particular circle’s stream page and read only the stuff shared by people in that circle. But when you comment on something there, unlike being in a cloistered room which this metaphor would have you believe, you’re actually talking to all the people whom the original author has decided to include into the conversation. You see them but you don’t really see them. It jars the mind a little bit. Will you really spend mental energy to keep track of where your comments will be seen? Not likely. So you either decide to ignore this “leak” with even more embarrassment potential than on Facebook, or you decide there is no privacy despite circles and revert, in behavioral terms, back to the Facebook model. Something is amiss.

That thing is the fact that there is no concept of “spacetime” online like in real life. Everything happens at one time and in one place despite efforts to counter it (RIAA discovered this). What you write can be read ten years later, and it can be transmitted everywhere, too. It has your identity attached to it as if you did it right then and there to all the people you have no control over.

Let’s reconsider social relationships. Circles is an attempt to model the different groups in which a person moves. These groups may be “socially independent” as Google researchers have found, but they are not disjoint. If they were or if they were nearly so, Google+ would have fully modeled the situation, so a “Colleague” circle and a “Family” circle may indeed be easy to keep track of (unless your coworker is also your brother). But once there are more than about 2-3 circles, things surely muddle up. So why do people feel the need to separate their communication to different groups in real life? Maybe it is less so about who are in a group, and more about the identity that the person wants to project in a group. The need for circles isn’t really one for categorizing other people, as much as one for categorizing your own identities. It is this need for compartmentalizing identities (code for “hiding” certain parts of a composite identity) that drives the need for privacy across group boundaries.

In my mind, a more powerful and frank model of social relationships recognizes this. It is a classic one that a generation of online users have already grown accustomed to. You shouldn’t have multiple circles for one identity, in the false hopes that they be disjoint. It defeats the whole purpose. You should have multiple disjoint identities, but easily accessible under one account. Maybe you have a “model child” identity to your family, an “industrious worker” identity to your colleagues, and I don’t know a “troll” (er, I mean “free-speech protected”) identity for your political rants. This probably requires complex per-item ACL, or simply pseudonymous identities. Some people cringe at this, but I don’t see why. Pseudonymous identities can be every bit as authentic as real ones. (Google+ has already run into a dilemma on this issue.) Moreover, each identity actually has to be accepted into your chosen cliques, so behavioral norms are already enforced intra-clique; identity labels are largely inconsequential. I think the replication thesis demands this since there is no spacetime disjointness online as exists in real life, so we can only go for identity disjointness. There are of course those (including Zuckerberg) who believe in radical unprivacy, and that is a utopian ideal, but for those opposed to it, Google+ unfortunately didn’t solve the problem.

(Why do “suit” and “purple” choose to stand?)

There is a better way to add capacity and reduce crowding in the subway car, however. Make the damned seats wider! What people do right now — due to some kind of human repelling force — is to skip every other seat unless the situation is really saturated, because it’s impolite and uncomfortable to squeeze next to/in between people. So MBTA ends up with half the designed seating capacity and the overflow just stand, sometimes right next to/blocking an empty seat. Just remove one seat per row and the number of seated people may increase dramatically. This applies to classrooms and other contexts, too.

Beginning a few years back, when graphics cards in computers not used for games became powerful enough to do something interesting, out came features that tried to take advantage of this power. I’m talking about things like circular windows, transparency, video wallpapers, flipping window previews, 3d desktops, like a cube or whatever…

Now trust me. Since I don’t play games, I’ve thought about using the idle GPU for other purposes, too. So I tried the above features. And they all suck. Greatly. (I also tried GPGPU but that’s a rant for another day.)

Along the way, I’ve become convinced that trying too hard to “use” GPU power is the wrong way to go. Alas, it’s probably better to stick to the basics and think of things that are actually useful. But first, why are the aforementioned graphics features so repugnant that users (at least I) turn them off after a few tries? Because they are unnatural, that’s why!

For example, the shape of windows. Look, rectangles are nice, for a good reason. They tesselate well. Hexagons may even work. Circles do not work. Odd shaped windows are the bane of my existence. How about the use of transparency? It never works! You don’t actually want to see what’s underneath, because if you do, it’s distracting. So transparency blurs what’s behind to the degree that you can’t tell what’s behind. This is paradoxical and completely useless. Video wallpapers. Bad idea. The desktop should not be distracting with movements. The eye is extremely capable of detecting scene change, especially change due to movement. You do not want the background to move at all! Flipping windows as in Vista… no. It’s harder to see what’s in the window due to the 3d angle and harder to see which one is at the forefront because of a stack of very different looking windows all competing for attention. Cube or spherical surfaces for desktops a la Compiz: fail. Nothing on our current desktop naturally wraps around a cube or sphere. You’re introducing extra distortion to generate the unnecessary projection to 2D. This is stupid.

However, I can think of two useful graphics features that are modifications of the failures described here:

1. Getting things out of your way
Do you see all these people with multiple monitors? Why? Because people want space. They want multiple windows showing at the same time. They want a big desktop. Now desktop resolutions even on a single monitor these days are by no means small. I argue that it is the extremely inefficient use of a single monitor that drives people to multiple monitors. This is quite ridiculous, actually. Why do I say this? Because I’ve seen a good MDI (multiple document interface) in, say, source code editors and development environments that can easily put all the documents you want (usually just two) on screen at the same time. So there is no reason to give up on the same level of efficiency between multiple applications.

Okay, so what am I talking about. I think most space problems can be solved if there is a way to get things out of your way. For example, if I want to tesselate a screen with rectangles I may end up leaving unused space due to some windows being certain sizes. Wouldn’t it be nice for windows to be unions of rectangles and their contents flow within them naturally so that the entire screen looked like a newspaper page layout? This requires window content to be amenable to reflow, but whatever. It can be done. Barring that, it’d be nice if you can throw windows out of your way to a automatically determined reasonable location, either temporarily or permanently. Windows 7 sort of does that (but only to pre-defined locations and sizes like half a screen) and Mac OS X sort ot does the … opposite … of that by bringing all windows into view…

2. Changing backgrounds
While video backgrounds are extremely distracting, a changing background itself isn’t a bad idea at all. The background can be equally used to get attention, as a conduit for conveying information — provided that is the goal. After all, it is already used to convey some information, with desktop icons, etc., although this happens only when the background is actually the foreground — when all other windows are out of the way. So can the background convey information while still being the background, and how?

Yes, it can. First thing to realize is the background can’t really convey a huge amount of information, since during normal operation, either (a) a small portion of it is revealed at a time — e.g. the gap between windows; and (2) a large portion is revealed only briefly — e.g. when switching windows. But a small amount of information can still be conveyed through this. One of the best uses for the background is to convey quasistatic or low frequency data. For instance, you’ve seen little weather and time indicators… why not let the background show the time and weather display by actually displaying a scene of that? It can be either current conditions or forecast, so long as the user knows what it is. So, if you want to know if it will rain 6 hours from now in Capetown, and have set the display to show that, the desktop can simply display a scene of that — and this doesn’t really change except when the weather changes, which is precisely when you want to be distracted with this new information, and in a non-intrusive way (you’ll just notice something new between switching windows). At other times you don’t even notice.

I’m sure there are other applications like these. There is no excuse to pretend that piss poor UI design is acceptable if it just seems “cool”.