Semi-analytical approximations to statistical moments of sigmoid and softmax mappings of normal variables

This note is concerned with accurate and computationally efficient approximations of moments of Gaussian random variables passed through sigmoid or softmax mappings. These approximations are semi-analytical (i.e. they involve the numerical adjustment of parametric forms) and highly accurate (they yield 5% error at most). We also highlight a few niche applications of these approximations, which arise in the context of, e.g., drift-diffusion models of decision making or non-parametric data clustering approaches. We provide these as examples of efficient alternatives to more tedious derivations that would be needed if one was to approach the underlying mathematical issues in a more formal way. We hope that this technical note will be helpful to modellers facing similar mathematical issues, although maybe stemming from different academic prospects.

[1]  Ivan Netuka,et al.  On threshold autoregressive processes , 1984, Kybernetika.

[2]  S. Domínguez-Almendros,et al.  Logistic regression models. , 2011, Allergologia et immunopathologia.

[3]  Felix Wichmann,et al.  The psychometric function: II. Bootstrap-based confidence intervals and sampling , 2001, Perception & psychophysics.

[4]  Karl J. Friston,et al.  Variational Bayesian identification and prediction of stochastic nonlinear dynamic causal models , 2009, Physica D. Nonlinear phenomena.

[5]  Karl J. Friston,et al.  A Bayesian Foundation for Individual Learning Under Uncertainty , 2011, Front. Hum. Neurosci..

[6]  Jean Daunizeau,et al.  The Social Bayesian Brain: Does Mentalizing Make a Difference When We Learn? , 2014, PLoS Comput. Biol..

[7]  Matthew J. Beal.  Variational algorithms for approximate Bayesian inference , 2003 .

[8]  M. W. Green,et al.  2. Handbook of the Logistic Distribution , 1991 .

[9]  Karl J. Friston,et al.  Observing the Observer (II): Deciding When to Decide , 2010, PloS one.

[10]  A. O'Hagan,et al.  Bayes estimation subject to uncertainty about parameter constraints , 1976 .

[11]  Karl J. Friston,et al.  Population dynamics: Variance and the sigmoid activation function , 2008, NeuroImage.

[12]  Amir Karniel,et al.  The exponential learning equation as a function of successful trials results in sigmoid performance , 2010 .

[13]  Gary R. Waissi,et al.  A sigmoid approximation of the standard normal integral , 1996 .

[14]  Karl J. Friston,et al.  Variational free energy and the Laplace approximation , 2007, NeuroImage.

[15]  R. Ratcliff,et al.  Connectionist and diffusion models of reaction time. , 1999, Psychological review.

[16]  Claudio Moraga,et al.  The Influence of the Sigmoid Function Parameters on the Speed of Backpropagation Learning , 1995, IWANN.

[17]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[18]  V. Šmídl,et al.  The Variational Bayes Method in Signal Processing , 2005 .

[19]  J. Daunizeau,et al.  Automatic integration of confidence in the brain valuation signal , 2015, Nature Neuroscience.

[20]  Anil K. Jain,et al.  Artificial Neural Networks: A Tutorial , 1996, Computer.

[21]  อนิรุธ สืบสิงห์,et al.  Data Mining Practical Machine Learning Tools and Techniques , 2014 .