Entropy, Information Theory, Information Geometry and Bayesian Inference in Data, Signal and Image Processing and Inverse Problems
Open Access
- 12 June 2015
- Vol. 17 (6), 3989-4027
- https://doi.org/10.3390/e17063989
Abstract
The main content of this review article is first to review the main inference tools using Bayes rule, the maximum entropy principle (MEP), information theory, relative entropy and the Kullback–Leibler (KL) divergence, Fisher information and its corresponding geometries. For each of these tools, the precise context of their use is described. The second part of the paper is focused on the ways these tools have been used in data, signal and image processing and in the inverse problems, which arise in different physical sciences and engineering applications. A few examples of the applications are described: entropy in independent components analysis (ICA) and in blind source separation, Fisher information in data model selection, different maximum entropy-based methods in time series spectral estimation and in linear inverse problems and, finally, the Bayesian inference for general inverse problems. Some original materials concerning the approximate Bayesian computation (ABC) and, in particular, the variational Bayesian approximation (VBA) methods are also presented. VBA is used for proposing an alternative Bayesian computational tool to the classical Markov chain Monte Carlo (MCMC) methods. We will also see that VBA englobes joint maximum a posteriori (MAP), as well as the different expectation-maximization (EM) algorithms as particular cases.This publication has 68 references indexed in Scilit:
- A tutorial on approximate Bayesian computationJournal of Mathematical Psychology, 2012
- The Bayesian LassoJournal of the American Statistical Association, 2008
- Variational Bayesian learning of directed graphical models with hidden variablesBayesian Analysis, 2006
- Fisher information inequalities and the central limit theoremProbability Theory and Related Fields, 2004
- Divergence measures based on the Shannon entropyIEEE Transactions on Information Theory, 1991
- Sampling-Based Approaches to Calculating Marginal DensitiesJournal of the American Statistical Association, 1990
- Linear inversion by the method of maximum entropyInverse Problems, 1989
- Image restoration by a powerful maximum entropy methodComputer Vision, Graphics, and Image Processing, 1983
- A new look at the statistical model identificationIEEE Transactions on Automatic Control, 1974
- Information Theory and Statistical MechanicsPhysical Review B, 1957