It’s about the link between computation and entropy. I take the idea of a Turing machine for granted, but starting with that I explain recursive functions, the Church-Turing thesis, Kolomogorov complexity, the relation between Kolmogorov complexity and Shannon entropy, the uncomputability of Kolmogorov complexity, the ‘complexity barrier’, Levin’s computable version of complexity, and finally my work with Mike Stay on algorithmic thermodynamics.

For more details, read our paper:

• John Baez and Mike Stay, Algorithmic thermodynamics, *Math. Struct. Comp. Sci.* **22** (2012), 771-787.

or these blog articles:

• Algorithmic thermodynamics (part 1).

• Algorithmic thermodynamics (part 2).

They all emphasize slightly different aspects!

]]>So, I’ll remove that quote, and hope the other one is really due to Shannon. Certainly Shannon had some ideas along these general lines…

]]>This is referred to as a quote from Shannon. I would be glad to find it in Shannon’s writings but I think it actually comes from Gregory Bateson. For instance, in Bateson, Gregory 1972. Steps to an Ecology of Mind. New York: Ballantine, he says: “A ‘bit’ of information is defined as a difference which makes a difference.” (p. 315), or in Bateson, Gregory 1979. Mind and nature: A necessary unity. New York: Dutton, he says: “Information consists of differences that make a difference.” (p. 110). ]]>

Maybe, but last time I did not understand what a “prefix-free Turing machine” is :-)

The definition is in the paper; I bet you just didn’t know why it’s interesting. And it’s actually not very interesting — it’s just a useful trick.

There are lots of ways to list programs. We say such a way is “prefix-free” if the name of one program that halts never appears as the initial part of the name of another one that halts. For example, you can’t have a program that halts named

01010

if you also have one named

01010110

It’s easy to accomplish this: just require that every program ends with a string that means “END”, say for example 1111, and require that this string only shows up at the end of the program. So, you might have a program named

0010101111

but then you can’t have one named

0010101111001111

None of this stuff is very important or profound. It’s just a standard trick for defining a “probability” for programs! Suppose the probability of a program is 2^{}^{-N} when its name has N bits. Then *if the naming system is prefix-free*, the sum of the probabilities of all programs that halt is ≤ 1. You can imagine flipping a coin until the bit string it generates is the name of a program, and the chance you got that sequence of coin flips is the measure of your program.

(There may also be cases where you keep flipping the coin and never get a name of a program that halts.)

(You’ll note that sometimes I say “program that halts” and sometimes I say “program”. Again, this is just technical baloney.)

]]>Perhaps part of the problem is that I’m trying to explain these concepts in everyday language, instead of using jargon like “prefix-free Turing machines”.

Maybe, but last time I did not understand what a “prefix-free Turing machine” is :-)

But this paper of Mike Stay helped me a lot in this regard:

Very Simple Chaitin Machines for Concrete AIT.