Fast ABC with joint generative modelling and subset simulation

We propose a novel approach for solving inverse-problems with high-dimensional inputs and an expensive forward mapping. It leverages joint deep generative modelling to transfer the original problem spaces to a lower dimensional latent space. By jointly modelling input and output variables and endowing the latent with a prior distribution, the fitted probabilistic model indirectly gives access to the approximate conditional distributions of interest. Since model error and observational noise with unknown distributions are common in practice, we resort to likelihood-free inference with Approximate Bayesian Computation (ABC). Our method calls on ABC by Subset Simulation to explore the regions of the latent space with dissimilarities between generated and observed outputs below prescribed thresholds. We diagnose the diversity of approximate posterior solutions by monitoring the probability content of these regions as a function of the threshold. We further analyze the curvature of the resulting diagnostic curve to propose an adequate ABC threshold. When applied to a cross-borehole tomography example from geophysics, our approach delivers promising performance without using prior knowledge of the forward nor of the noise distribution.

[1]  Lukas Mosser,et al.  Rapid seismic domain transfer: Seismic velocity inversion and modeling using deep generative neural networks , 2018, 80th EAGE Conference and Exhibition 2018.

[2]  Hoon Kim,et al.  Monte Carlo Statistical Methods , 2000, Technometrics.

[3]  Guillermo Rus-Carlborg,et al.  Approximate Bayesian Computation by Subset Simulation , 2014, SIAM J. Sci. Comput..

[4]  Ullrich Köthe,et al.  Benchmarking Invertible Architectures on Inverse Problems , 2021, ArXiv.

[5]  Nicholas Zabaras,et al.  Solving inverse problems using conditional invertible neural networks , 2020, J. Comput. Phys..

[6]  Michael U. Gutmann,et al.  Adaptive Gaussian Copula ABC , 2019, AISTATS.

[7]  J. Marin,et al.  Adaptivity for ABC algorithms: the ABC-PMC scheme , 2008 .

[8]  Olivier François,et al.  Non-linear regression models for Approximate Bayesian Computation , 2008, Stat. Comput..

[9]  Richard E. Turner,et al.  Neural Adaptive Sequential Monte Carlo , 2015, NIPS.

[10]  Eric Laloy,et al.  Approaching geoscientific inverse problems with adversarial vector-to-image domain transfer networks , 2019 .

[11]  Ahmed H. Elsheikh,et al.  Accelerating Monte Carlo Markov chains with proxy and error models , 2015, Comput. Geosci..

[12]  Daniele Faccio,et al.  Variational Inference for Computational Imaging Inverse Problems , 2019, J. Mach. Learn. Res..

[13]  Bernhard Schölkopf,et al.  Wasserstein Auto-Encoders , 2017, ICLR.

[14]  Ullrich Köthe,et al.  Analyzing Inverse Problems with Invertible Neural Networks , 2018, ICLR.

[15]  Yuichi Yoshida,et al.  Spectral Normalization for Generative Adversarial Networks , 2018, ICLR.

[16]  Jonas Adler,et al.  Deep Bayesian Inversion , 2018, ArXiv.

[17]  Ali Razavi,et al.  Preventing Posterior Collapse with delta-VAEs , 2019, ICLR.

[18]  Martin J. Blunt,et al.  Stochastic Seismic Waveform Inversion Using Generative Adversarial Networks as a Geological Prior , 2018, Mathematical Geosciences.

[19]  Honglak Lee,et al.  Learning Structured Output Representation using Deep Conditional Generative Models , 2015, NIPS.

[20]  Philippe Renard,et al.  Using Generative Adversarial Networks as a Fast Forward Operator for Hydrogeological Inverse Problems , 2020, Ground water.

[21]  Miss A.O. Penney (b) , 1974, The New Yale Book of Quotations.

[22]  Yalchin Efendiev,et al.  An efficient two‐stage Markov chain Monte Carlo method for dynamic data integration , 2005 .

[23]  Daniela Calvetti,et al.  Inverse problems: From regularization to Bayesian inference , 2018 .

[24]  Gabriel Peyré,et al.  Learning Generative Models with Sinkhorn Divergences , 2017, AISTATS.

[25]  Michael Unser,et al.  Convolutional Neural Networks for Inverse Problems in Imaging: A Review , 2017, IEEE Signal Processing Magazine.

[26]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[27]  William J. Welch,et al.  CcGAN: Continuous Conditional Generative Adversarial Networks for Image Generation , 2020, ICLR.

[28]  Paul Marjoram,et al.  Markov chain Monte Carlo without likelihoods , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[29]  Aki Vehtari,et al.  Gaussian process modelling in approximate Bayesian computation to estimate horizontal gene transfer in bacteria , 2016, The Annals of Applied Statistics.

[30]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[31]  Iain Murray,et al.  Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows , 2018, AISTATS.

[32]  Aggelos K. Katsaggelos,et al.  Using Deep Neural Networks for Inverse Problems in Imaging: Beyond Analytical Methods , 2018, IEEE Signal Processing Magazine.

[33]  Eric Laloy,et al.  Deep generative models in inversion: The impact of the generator's nonlinearity and development of a new approach based on a variational autoencoder , 2021, Comput. Geosci..

[34]  Albert Tarantola,et al.  Inverse problem theory - and methods for model parameter estimation , 2004 .

[35]  Christian P. Robert,et al.  Accelerating MCMC algorithms , 2018, Wiley interdisciplinary reviews. Computational statistics.

[36]  Michael U. Gutmann,et al.  Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models , 2015, J. Mach. Learn. Res..

[37]  Alexandros G. Dimakis,et al.  Deep Learning Techniques for Inverse Problems in Imaging , 2020, IEEE Journal on Selected Areas in Information Theory.

[38]  J. Beck,et al.  Estimation of Small Failure Probabilities in High Dimensions by Subset Simulation , 2001 .

[39]  Iason Papaioannou,et al.  MCMC algorithms for Subset Simulation , 2015 .

[40]  Lambros S. Katafygiotis,et al.  Bayesian post-processor and other enhancements of Subset Simulation for estimating failure probabilities in high dimensions , 2011 .

[41]  Ivan Kobyzev,et al.  Normalizing Flows: An Introduction and Review of Current Methods , 2020, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Diederik P. Kingma,et al.  An Introduction to Variational Autoencoders , 2019, Found. Trends Mach. Learn..

[43]  Jordan M. Malof,et al.  Benchmarking deep inverse models over time, and the neural-adjoint method , 2020, NeurIPS.

[44]  Yanan Fan,et al.  Handbook of Approximate Bayesian Computation , 2018 .

[45]  Kyungwoo Song,et al.  Sequential Likelihood-Free Inference with Implicit Surrogate Proposal , 2020, ArXiv.

[46]  Tom White,et al.  Generative Adversarial Networks: An Overview , 2017, IEEE Signal Processing Magazine.

[47]  David S. Greenberg,et al.  Automatic Posterior Transformation for Likelihood-Free Inference , 2019, ICML.

[48]  Aki Vehtari,et al.  Parallel Gaussian Process Surrogate Bayesian Inference with Noisy Likelihood Evaluations , 2019, Bayesian Analysis.

[49]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[50]  Eric Laloy,et al.  Inversion using a new low-dimensional representation of complex binary geological media based on a deep neural network , 2017, 1710.09196.