Rényi Divergence and Kullback-Leibler Divergence
Top Cited Papers
- 12 June 2014
- journal article
- Published by Institute of Electrical and Electronics Engineers (IEEE) in IEEE Transactions on Information Theory
- Vol. 60 (7), 3797-3820
- https://doi.org/10.1109/tit.2014.2320500
Abstract
Rényi divergence is related to Rényi entropy much like Kullback-Leibler divergence is related to Shannon's entropy, and comes up in many settings. It was introduced by Rényi as a measure of information that satisfies almost the same axioms as Kullback-Leibler divergence, and depends on a parameter that is called its order. In particular, the Rényi divergence of order 1 equals the Kullback-Leibler divergence. We review and extend the most important properties of Rényi divergence and Kullback-Leibler divergence, including convexity, continuity, limits of \(\sigma \) -algebras, and the relation of the special order 0 to the Gaussian dichotomy and contiguity. We also show how to generalize the Pythagorean inequality to orders different from 1, and we extend the known equivalence between channel capacity and minimax redundancy to continuous channel inputs (for all orders) and present several other minimax results.
Keywords
Other Versions
Funding Information
- Netherlands Organization for Scientific Research through the Rubicon Programme (680-50-1112)
This publication has 42 references indexed in Scilit:
- Rényi divergence measures for commonly used univariate continuous distributionsInformation Sciences, 2013
- From ɛ-entropy to KL-entropy: Analysis of minimum information complexity density estimationThe Annals of Statistics, 2006
- On Choosing and Bounding Probability MetricsInternational Statistical Review, 2002
- Hellinger-Consistency of Certain Nonparametric Maximum Likelihood EstimatorsThe Annals of Statistics, 1993
- Fisher Information and Dichotomies in Equivalence/ContiguityThe Annals of Probability, 1989
- Elementary proof for Sion's minimax theoremKodai Mathematical Journal, 1988
- Renyi's entropy and the probability of errorIEEE Transactions on Information Theory, 1978
- Equivalence and perpendicularity of Gaussian processesPacific Journal of Mathematics, 1958
- On general minimax theoremsPacific Journal of Mathematics, 1958
- On Equivalence of Infinite Product MeasuresAnnals of Mathematics, 1948