I hope I have enough energy to say more about the asymptotic equipartition property.

By the way, Jamie, speaking of error correction…

Everyone please remember: on all WordPress blogs, LaTeX is done like this:

$latex E = mc^2$

with the word ‘latex’ directly following the first dollar sign, no space. Double dollar signs don’t work here.

I stated a watered-down version of the asymptotic equipartition theorem: just for purposes of exposition, I assumed that each letter in the string was drawn independently from the same probability distribution on letters. In other words, I was assuming that they’re ‘independent identically distributed’ random variables. This is clearly too restrictive—it sure ain’t true for English text!

The statement and proof gets a bit harder when we do the full-fledged thing: you can see a proof here for the i.i.d. case and here for the more general case. The more general case uses Lévy’s martingale convergence theorem, Markov’s inequality and Borel-Cantelli lemma. For some people that would be very scary, while others eat such stuff for breakfast.

But anyway, regardless of whether the proof is hard, the result seems both interesting and highly believable: not shocking.

My quick skim of your summary of the hard lemma left me wondering why it is a hard lemma*. Isn’t it just a baby step away from the fact that as n-> infinity, samples of size n look more and more like the underlying distribution from which they are drawn?

*That in itself is progress, since my quick skim of Wikipedia’s summary of the hard lemma left me almost clueless as to the content of the hard lemma (it struck me as very vague:-)

This is interesting with respect to the post “Mathematics of biodiversity 7” to understand the whole concept of bias correction of entropy estimates in the case of a limited sample.

Consider the quantity

where is a sequence of letters occurring with probabilities . And let’s assume there are different letters..

In the case where the letters are i.i.d., then

Now, the same may appear many times, say , so this formula can be rewritten

so that:

When is large, then

QED.

I wonder if there are classical analogues of these sorts of result.

Tobias, I agree, it’s really exciting to think about how all this stuff generalizes to topologically interesting situations. It’s entrancing to see linear algebra coming into play in the butterfly network.