BTW, we have yet another discrete entropy estimator that doesn’t assume . It’s a nonparametric Bayesian estimator based on a Pitman-Yor process prior.

Paper: Evan Archer*, Il Memming Park*, Jonathan Pillow. Bayesian Entropy Estimation for Countable Discrete Distributions. arXiv:1302.0328 (2013)

Code available: https://github.com/pillowlab/PYMentropy

]]>Interesting, thanks! I’ll have to get back to this subject sometime… I’ve been distracted by other projects.

]]>I just wanted to add that the paper by myself and Schürmann has a second estimator, where you don’t need to observe every symbol at least once.

But it doesn’t violate Prop 8 of Liam Paninski’s paper. Prop 8 requires that the sample size be given in advance, whereas our estimators have no a priori bound on the sample size.

]]>Thanks, I’ll fix this. I’ll be interested to hear what you’re thinking about.

]]>Thanks! I believe the new version of my blog article gets the facts straight—if you see any actual errors please let me know.

I believe the usual definition of ‘estimator’ requires that you’re able to compute an estimate from samples of data no matter what those samples are. As you note, Smith and Schürmann’s result evades Paninski’s theorem by relaxing this definition of ‘estimator’: their formula only lets you compute an estimate if your samples obey some condition. One might call this a ‘conditional estimator’. I imagine some statisticians have already thought hard about this idea.

]]>Actually I’ve seen the same (or very similar) entropy estimator as in the Smith and Schürmann paper you mentioned. And the unbiasedness proof seems to be right, given that we have enough data to observe every symbol at least once (for ) while Paninski’s proof assumes a fixed number of symbols observed. Given infinite data, most entropy estimators are in fact asymptotically unbiased.

BTW, if the number of symbols (or species) is known, entropy is always bounded by , so the variance cannot be infinite. When it is unknown, it is a different story.

]]>Typo alert: $X$ is missing its “latex”.

(I had a more substantial comment to make here, but I realised I should explicitly work through the de-Finetti-theorem-related stuff I was thinking about before I started yapping on over it.)

]]>The most recent thing I’ve seen in the tradition I was talking about is this:

A. Bialas, W. Czyz, K. Zalewski (2006), “Measurement of Renyi entropies in multiparticle production: a DO-LIST II” *Acta Physica Polonica B* **37:** 2713–28 [arXiv:hep-ph/0607082].