I believe the usual definition of ‘estimator’ requires that you’re able to compute an estimate from samples of data no matter what those samples are. As you note, Smith and Schürmann’s result evades Paninski’s theorem by relaxing this definition of ‘estimator’: their formula only lets you compute an estimate if your samples obey some condition. One might call this a ‘conditional estimator’. I imagine some statisticians have already thought hard about this idea.

BTW, if the number of symbols (or species) is known, entropy is always bounded by , so the variance cannot be infinite. When it is unknown, it is a different story.

(I had a more substantial comment to make here, but I realised I should explicitly work through the de-Finetti-theorem-related stuff I was thinking about before I started yapping on over it.)

A. Bialas, W. Czyz, K. Zalewski (2006), “Measurement of Renyi entropies in multiparticle production: a DO-LIST II” Acta Physica Polonica B 37: 2713–28 [arXiv:hep-ph/0607082].

John, I just gave a talk yesterday afternoon here in Barcelona (we all miss you!!!) about estimation of species richness, entropy, and Hill numbers. I suggested a way to “tune” the observed relative abundances (which sum to unity) by noting that their true values must sum to the coverage deficit , where is the Good–Turing coverage, the population share of the species detected in the sample. The coverage deficit can be estimated from the sample as where is the number of singletons in the sample and n is sample size.

Chao and Shen use this method to tune all observed frequencies by a correction factor However, I later realized (and Anne agreed) that this is not quite correct (though it works very well anyway), because the relative uncertainty in the estimate of is very small for the large but is large for the small For example, a species represented only once in the sample, with frequency could easily have been missed or been found twice; its true frequency in the population might easily be or so our relative uncertainty in its value is huge. Contrast this with a species which makes up 50% of the sample. Using the binomial formula for standard deviation, we can show that the precision of this estimate is very high if is large. We don’t have the right to tune that frequency very much. We should tune all frequencies in proportion to their uncertainty. It turns out to be easy to write a simple formula for this.

Tuning only solves half the problem, though. In the case of entropy, the abundance distribution of the unseen species has a largish effect on the value of entropy. That part still needs more work. Hopefully Anne’s innovation will make this unnecessary.

Of course there’s a huge debate on what probabilities actually mean, with subjective Bayesians religiously avoiding phrases like “the true probability”—and for good reasons. It’s an important debate that I’ve spent a lot of time on. I’m actually a subjective Bayesian with mild reservations concerning the possibility that even this position doesn’t seem to get to the bottom of certain issues.

But here I was trying to explain the concept of an unbiased estimator without getting too technical. This is a mathematical issue that largely sidesteps the debate about what probabilities mean. But the problem is that when you talk using words, it’s hard to avoid seeming like you’re taking a position on that debate—and it’s a lot simpler to talk like someone who believes probabilities are an objective feature of reality.

So, in anything I write, you should be able to take phrases like “the true probability distribution” and replace them with “the probability distribution that models Fred’s beliefs” without anything bad happening.

Allowing myself to get more technical, the math goes like this:

Suppose is a probability distribution on a finite set . Its entropy is

Define an estimator for entropy to be a function

The idea is that given samples from the set $X$, the estimator gives a number that’s supposed to be an estimate of entropy. If these samples are independent and distributed according to the distribution , the mean estimated entropy will be

The bias of the estimator is the difference between the mean estimated entropy and the actual entropy of :

The estimator is unbiased if the bias is zero for all .

Proposition 8 of Paninski’s paper says there exists no unbiased estimator for entropy.

People often think of the ‘true’ probability distribution, whose entropy the estimator is seeking to estimate on the basis of samples distributed according to that distribution. But the math doesn’t care about the word ‘true’, and since the definition of ‘unbiased estimator’ involves a property that’s supposed to hold for all , we don’t even need any way to specify a particular one.