LJKProbability & Statistics Seminar

On Thursday June 22 2017 at 14h00 in Room 106  IMAG Building

Seminary of Guillaume DEHAENE (EPFL)

ExpectationPropagation performs smooth gradient descent

Summary

In most applications of Bayesian inference, the key object: the posterior probability distribution, is uncomputable. One possibility to deal with this issue consists in computing a parametric (e.g: Gaussian) approximation of the true uncomputable distribution, for example using Expectation Propagation (EP, Minka, 2001) or Stochastic Variational Inference (SVI, Hoffman, 2013). Both proceed by iteratively improving a parametric approximation of the true posterior distribution until a fixedpoint is reached.
Both of these algorithms have now existed for some time, but they are still slightly poorly understood. In this talk, I will shed new intuitive light on the behavior of the sequence of approximations of EP and SVI when they are used to compute a Gaussian approximation. I will do this by linking them both to gradient descent (more precisely, Newton's method: gradient descent with a Hessian correction). I will show that both EP and SVI can be understood as performing gradient descent on a smoothed version of the energy landscape.
