Convolutional Analysis Operator Learning: Dependence on Training Data

Convolutional analysis operator learning (CAOL) enables the unsupervised training of (hierarchical) convolutional sparsifying operators or autoencoders from large datasets. One can use many training images for CAOL, but a precise understanding of the impact of doing so has remained an open question. This letter presents a series of results that lend insight into the impact of dataset size on the filter update in CAOL. The first result is a general deterministic bound on errors in the estimated filters, and is followed by a bound on the expected errors as the number of training samples increases. The second result provides a high probability analogue. The bounds depend on properties of the training data, and we investigate their empirical values with real data. Taken together, these results provide evidence for the potential benefit of using more training data in CAOL.

[1]  Jian-Feng Cai,et al.  Data-driven tight frame construction and image denoising , 2014 .

[2]  Weiwei Sun,et al.  New Perturbation Bounds for Unitary Polar Factors , 2003, SIAM J. Matrix Anal. Appl..

[3]  Jeffrey A. Fessler,et al.  Convergent convolutional dictionary learning using Adaptive Contrast Enhancement (CDL-ACE): Application of CDL to image denoising , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[4]  Yoram Bresler,et al.  $\ell_{0}$ Sparsifying Transform Learning With Efficient Optimal Updates and Convergence Guarantees , 2015, IEEE Transactions on Signal Processing.

[5]  Barnabás Póczos,et al.  Minimax Reconstruction Risk of Convolutional Sparse Dictionary Learning , 2018, AISTATS.

[6]  Jeffrey A. Fessler,et al.  Deep BCD-Net Using Identical Encoding-Decoding CNN Structures for Iterative Image Recovery , 2018, 2018 IEEE 13th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP).

[7]  H. Weyl.  Das asymptotische Verteilungsgesetz der Eigenwerte linearer partieller Differentialgleichungen (mit einer Anwendung auf die Theorie der Hohlraumstrahlung) , 1912 .

[8]  Klaus Diepold,et al.  Analysis Operator Learning and its Application to Image Reconstruction , 2012, IEEE Transactions on Image Processing.

[9]  Ameet Talwalkar,et al.  Foundations of Machine Learning , 2012, Adaptive computation and machine learning.

[10]  Hans C. van Houwelingen,et al.  The Elements of Statistical Learning, Data Mining, Inference, and Prediction. Trevor Hastie, Robert Tibshirani and Jerome Friedman, Springer, New York, 2001. No. of pages: xvi+533. ISBN 0‐387‐95284‐5 , 2004 .

[11]  A. Bruckstein,et al.  K-SVD : An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation , 2005 .

[12]  Rémi Gribonval,et al.  Learning Co-Sparse Analysis Operators With Separable Structures , 2015, IEEE Transactions on Signal Processing.

[13]  Rémi Gribonval,et al.  Constrained Overcomplete Analysis Operator Learning for Cosparse Signal Modelling , 2012, IEEE Transactions on Signal Processing.

[14]  Robert Tibshirani,et al.  The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd Edition , 2001, Springer Series in Statistics.

[15]  Ren-Cang Li,et al.  New Perturbation Bounds for the Unitary Polar Factor , 1995, SIAM J. Matrix Anal. Appl..

[16]  Jeffrey A. Fessler,et al.  Momentum-Net: Fast and Convergent Iterative Neural Network for Inverse Problems , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[17]  Jeffrey A. Fessler,et al.  Convolutional analysis operator learning: Application to sparse-view CT : (Invited Paper) , 2018, 2018 52nd Asilomar Conference on Signals, Systems, and Computers.

[18]  Holger Rauhut,et al.  A Mathematical Introduction to Compressive Sensing , 2013, Applied and Numerical Harmonic Analysis.

[19]  Jeffrey A. Fessler,et al.  Convolutional Dictionary Learning: Acceleration and Convergence , 2017, IEEE Transactions on Image Processing.

[20]  J. Franklin,et al.  The elements of statistical learning: data mining, inference and prediction , 2005 .

[21]  Jeffrey A. Fessler,et al.  Fast and convergent iterative image recovery using trained convolutional neural networks , 2018, 2018 56th Annual Allerton Conference on Communication, Control, and Computing (Allerton).

[22]  Jeffrey A. Fessler,et al.  Convolutional Analysis Operator Learning: Acceleration and Convergence , 2018, IEEE Transactions on Image Processing.

[23]  M. Elad,et al.  $rm K$-SVD: An Algorithm for Designing Overcomplete Dictionaries for Sparse Representation , 2006, IEEE Transactions on Signal Processing.