http://en.wikipedia.org/wiki/Noisy-channel_coding_theorem

could have left it at the classical statement of the theorem with bullet #1. Then it goes on to say:

2. If a probability of bit error \(p_b\) is acceptable, rates up to \(R(p_b)\) are achievable, where

\(R(p_b) = \frac{C}{1-H_2(p_b)}\).

3. For any \(p_b\), rates greater than \(R(p_b)\) are not achievable.

I have never seen this before. At first glance, this seems questionable, as Fano’s converse gives \(P_e^{(n)} \ge 1 – \frac{1}{nR} – \frac{C}{R}\), which seems to converge to \(H_b(p_e) \ge p_e\) for \(p_e \in [0,0.5]\). So it must mean whatever is used to code this is not going to be a long block code.

One example where this is true is the binary symmetric channel, with uncoded transmission. But I’m not so sure what is the achievability scheme in general, although I have some ideas — it may involve quantizing the excess codewords to the nearest zero-error codewords. The converse I have no idea.

In terms of the statement, it is really unclear what is meant by “bit error”. In the classical statement, a message from a large alphabet is coded into some \(X^n \in \mathcal{X}^n\) where \(\mathcal{X}\) is the channel input alphabet. After decoding, \(X^n\) is either found correctly, or it is in error. There is no “bit” in here. Even if \(X\) is binary, is the bit error the received (uncooked) bit error? Or is it the decoded (cooked) bit error? Why should the decoded bit error matter, isn’t that a codebook artifact? Or is it the bit error in the original message, if the original message is to be represented by a bit-stream? But that is also entirely arbitrary.

Anyway I’d like a clarification from someone or a reference.

The problem: \(X, Y, Z\) are independent random variables uniformly distributed over [0,1]. What is the distribution of \((XY)^Z\)?

The conventional solution is to find the distribution of \(XY\) first, then of \((XY)^Z\).

The distribution of \(XY\) can be derived from its CDF \(F_{XY}(xy \le k)\), which is the total shaded area shown. The red curve is the function \(y=\frac{k}{x}\). This area is thus given by:

\(\int_k^1 \frac{k}{x} dx + k = k \ln(x) \vert_k^1 + k = -k \ln(k) + k\), for \(k>0\).

The PDF is the derivative of the above:

\(f_{XY}(k) = -\ln(k)\), for \(k>0\). The density is not well defined at \(k=0\).

The second part is to find the distribution in question. Here, the red curve is the function \(z=\frac{\ln(k)}{\ln(xy)}\). The CDF \(F_{(XY)^Z}((xy)^z \le k)\) is the total area to the left of the red curve. A column slice shaded in blue has probability per unit of \(z\) as labeled. Thus, the CDF is:

\(\int_0^k f_{XY}(v) dv (1-\frac{\ln(k)}{\ln(v)}) = \int_0^k -\ln(v) dv (1-\frac{\ln(k)}{\ln(v)})\)
\( = \int_0^k -\ln(v) dv + \int_0^k \ln(k) dv\)
\( = -v \ln(v) + v \vert_{\downarrow 0}^k + k \ln(k) = -k \ln(k) + k + k \ln(k) = k\), for \(k>0\).

For \(k=0\), we can fill in 0, because the minimum value of \((XY)^Z\) is 0, so it is the minimum of the support of its CDF.

So amazingly, \((XY)^Z\) has the CDF of (and is) a uniform distribution. How does that happen? What’s the intuition?

First post and already TeX can be rendered. I stole the idea from fakalin.

\( \int_{0}^{1}\frac{x^{4}\left( 1-x\right) ^{4}}{1+x^{2}}dx = \frac{22}{7}-\pi \)