Connections between Operator-splitting Methods and Deep Neural Networks with Applications in Image Segmentation

Deep neural network is a powerful tool for many tasks. Understanding why it is so successful and providing a mathematical explanation is an important problem and has been one popular research direction in past years. In the literature of mathematical analysis of deep deep neural networks, a lot of works are dedicated to establishing representation theories. How to make connections between deep neural networks and mathematical algorithms is still under development. In this paper, we give an algorithmic explanation for deep neural networks, especially in their connection with operator splitting and multigrid methods. We show that with certain splitting strategies, operator-splitting methods have the same structure as networks. Utilizing this connection and the Potts model for image segmentation, two networks inspired by operator-splitting methods are proposed. The two networks are essentially two operator-splitting algorithms solving the Potts model. Numerical experiments are presented to demonstrate the effectiveness of the proposed networks.

[1]  Hao Liu,et al.  PottsMGNet: A Mathematical Explanation of Encoder-Decoder Based Neural Networks , 2023, ArXiv.

[2]  Rongjie Lai,et al.  Deep Nonparametric Estimation of Intrinsic Data Structures by Chart Autoencoders: Generalization Error and Robustness , 2023, ArXiv.

[3]  Z. Li,et al.  DOSnet as a Non-Black-Box PDE Solver: When Deep Learning Meets Operator Splitting , 2022, J. Comput. Phys..

[4]  Xinlong Feng,et al.  An efficient maximum bound principle preserving p-adaptive operator-splitting method for three-dimensional phase field shape transformation model , 2022, Comput. Math. Appl..

[5]  R. Glowinski,et al.  A Fast Operator-splitting Method for Beltrami Color Image Denoising , 2022, Journal of Scientific Computing.

[6]  Wenjing Liao,et al.  Benefits of Overparameterized Convolutional Residual Networks: Function Approximation under Smoothness Constraint , 2022, ICML.

[7]  A. Caboussat,et al.  A splitting method for the numerical simulation of free surface flows with sediment deposition and resuspension , 2022, International Journal for Numerical Methods in Fluids.

[8]  R. Glowinski,et al.  Elastica Models for Color Image Regularization , 2022, SIAM J. Imaging Sci..

[9]  Hao Liu,et al.  Fast operator splitting methods for obstacle problems , 2022, J. Comput. Phys..

[10]  Wenjing Liao,et al.  Besov Function Approximation and Binary Classification on Low-Dimensional Manifolds Using Convolutional Residual Networks , 2021, ICML.

[11]  R. Glowinski,et al.  An Operator-Splitting Method for the Gaussian Curvature Regularization Model with Applications in Surface Smoothing and Imaging , 2021, SIAM J. Sci. Comput..

[12]  George Em Karniadakis,et al.  Error estimates for DeepOnets: A deep learning framework in infinite dimensions , 2021, Transactions of Mathematics and Its Applications.

[13]  Xue-Cheng Tai,et al.  Efficient and Convergent Preconditioned ADMM for the Potts Models , 2021, SIAM J. Sci. Comput..

[14]  Roland Glowinski,et al.  A Color Elastica Model for Vector-Valued Image Regularization , 2020, SIAM J. Imaging Sci..

[15]  Jun Liu,et al.  Deep Convolutional Neural Networks with Spatial Regularization, Volume and Star-Shape Priors for Image Segmentation , 2020, Journal of Mathematical Imaging and Vision.

[16]  Ding-Xuan Zhou,et al.  Theory of deep convolutional neural networks: Downsampling , 2020, Neural Networks.

[17]  Hao Liu,et al.  Curvature Regularized Surface Reconstruction from Point Cloud , 2020, SIAM J. Imaging Sci..

[18]  Zuowei Shen,et al.  Deep Network Approximation for Smooth Functions , 2020, SIAM J. Math. Anal..

[19]  Taiji Suzuki,et al.  Approximation and Non-parametric Estimation of ResNet-type Convolutional Neural Networks , 2019, ICML.

[20]  Xue-Cheng Tai,et al.  A New Operator Splitting Method for the Euler Elastica Model for Image Smoothing , 2018, SIAM J. Imaging Sci..

[21]  Nima Tajbakhsh,et al.  UNet++: A Nested U-Net Architecture for Medical Image Segmentation , 2018, DLMIA/ML-CDS@MICCAI.

[22]  Ding-Xuan Zhou,et al.  Universality of Deep Convolutional Neural Networks , 2018, Applied and Computational Harmonic Analysis.

[23]  Raymond H. Chan,et al.  Convex non-convex image segmentation , 2018, Numerische Mathematik.

[24]  Wotao Yin,et al.  Splitting Methods in Communication, Imaging, Science, and Engineering , 2017 .

[25]  Maryam Yashtini,et al.  A Fast Relaxed Normal Two Split Method and an Effective Weighted TV Approach for Euler's Elastica Image Inpainting , 2016, SIAM J. Imaging Sci..

[26]  Dmitry Yarotsky,et al.  Error bounds for approximations with deep ReLU networks , 2016, Neural Networks.

[27]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Roland Glowinski,et al.  A Penalization-Regularization-Operator Splitting Method for Eikonal Based Traveltime Tomography , 2015, SIAM J. Imaging Sci..

[29]  Raymond H. Chan,et al.  A Three-Stage Approach for Segmenting Degraded Color Images: Smoothing, Lifting and Thresholding (SLaT) , 2015, J. Sci. Comput..

[30]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[31]  Raymond H. Chan,et al.  A Two-Stage Image Segmentation Method for Blurry Images with Poisson or Multiplicative Gamma Noise , 2014, SIAM J. Imaging Sci..

[32]  Annalisa Quaini,et al.  Fluid-structure interaction in blood flow capturing non-zero longitudinal structure displacement , 2012, J. Comput. Phys..

[33]  Shi-Min Hu,et al.  Global contrast based salient region detection , 2011, CVPR 2011.

[34]  Antonin Chambolle,et al.  A First-Order Primal-Dual Algorithm for Convex Problems with Applications to Imaging , 2011, Journal of Mathematical Imaging and Vision.

[35]  Xue-Cheng Tai,et al.  Global Minimization for Continuous Multiphase Partitioning Problems Using a Dual Approach , 2011, International Journal of Computer Vision.

[36]  Xue-Cheng Tai,et al.  A Continuous Max-Flow Approach to Potts Model , 2010, ECCV.

[37]  Xue-Cheng Tai,et al.  A study on continuous max-flow and min-cut approaches , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Vladimir Kolmogorov,et al.  An experimental comparison of min-cut/max- flow algorithms for energy minimization in vision , 2001, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[39]  L. Vese,et al.  Active contours without edges , 2001, IEEE Trans. Image Process..

[40]  Tony F. Chan,et al.  An Active Contour Model without Edges , 1999, Scale-Space.

[41]  R. Kimmel,et al.  Geodesic Active Contours , 1995, Proceedings of IEEE International Conference on Computer Vision.

[42]  Andrew R. Barron,et al.  Universal approximation bounds for superpositions of a sigmoidal function , 1993, IEEE Trans. Inf. Theory.

[43]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[44]  Lawrence D. Jackel,et al.  Backpropagation Applied to Handwritten Zip Code Recognition , 1989, Neural Computation.

[45]  D. Mumford,et al.  Optimal approximations by piecewise smooth functions and associated variational problems , 1989 .

[46]  L. Modica.  The gradient theory of phase transitions and the minimal interface criterion , 1987 .

[47]  J. Marsden,et al.  Product formulas and numerical algorithms , 1978 .

[48]  G. Strang.  On the Construction and Comparison of Difference Schemes , 1968 .

[49]  R. B. Potts.  Some generalized order-disorder transformations , 1952, Mathematical Proceedings of the Cambridge Philosophical Society.

[50]  Xue-Cheng Tai,et al.  The Potts Model with Different Piecewise Constant Representations and Fast Algorithms: A Survey , 2021 .

[51]  Yan Li,et al.  MA-Net: A Multi-Scale Attention Network for Liver and Tumor Segmentation , 2020, IEEE Access.

[52]  Xuecheng Tai,et al.  Augmented Lagrangian method for an Euler's elastica based segmentation model that promotes convex contours , 2017 .

[53]  Xue-Cheng Tai,et al.  Some Facts About Operator-Splitting and Alternating Direction Methods , 2016 .

[54]  Alexandre Caboussat,et al.  Operator Splitting Algorithms for Free Surface Flows: Application to Extrusion Processes , 2016 .

[55]  Xue-Cheng Tai,et al.  Primal-dual method for continuous max-flow approaches , 2015 .

[56]  G. Strang,et al.  Operator splitting , 2011 .

[57]  D. Pallara,et al.  Short-time heat flow and functions of bounded variation in RN , 2005 .

[58]  Demetri Terzopoulos,et al.  Snakes: Active contour models , 2004, International Journal of Computer Vision.

[59]  Xuecheng Tai,et al.  A parallel splitting-up method for partial differential equations and its applications to Navier-Stokes equations , 1992 .

[60]  G. Marchuk.  Splitting and alternating direction methods , 1990 .

[61]  Andrew M. Stuart,et al.  SMAI-JCM SMAI Journal of Computational Mathematics Model Reduction And Neural Networks For Parametric PDEs , 2022 .