Deep Learning and Higher Degree F-Transforms: Interpretable Kernels Before and After Learning

One of the current trends in the deep neural network technology consists in allowing a man–machine interaction and providing an explanation of network design and learning principles. In this direction, an experience with fuzzy systems is of great support. We propose our insight that is based on the particular theory of fuzzy (F)-transforms. Besides a theoretical explanation, we develop a new architecture of a deep neural network where the F-transform convolution kernels are used in the first two layers. Based on a series of experiments, we demonstrate the suitability of the F-transform-based deep neural network in the domain of image processing with the focus on recognition. Moreover, we support our insight by revealing the similarity between the F-transform and first-layer kernels in the most used deep neural networks.

[1]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[2]  Irina Perfilieva,et al.  Total variation with nonlocal FT-Laplacian for patch-based inpainting , 2018, Soft Comput..

[3]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[4]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[5]  Vilém Novák,et al.  Analysis and prediction of time series using fuzzy transform , 2008, 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence).

[6]  Irina Perfilieva,et al.  Convolutional Neural Networks with the F-transform Kernels , 2017, IWANN.

[7]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[8]  Kuntal Kumar Pal,et al.  Preprocessing for image classification by convolutional neural networks , 2016, 2016 IEEE International Conference on Recent Trends in Electronics, Information & Communication Technology (RTEICT).

[9]  Joan Bruna,et al.  Mathematics of Deep Learning , 2017, ArXiv.

[10]  Omolbanin Yazdanbakhsh,et al.  A Deep Neuro-Fuzzy Network for Image Classification , 2020, ArXiv.

[11]  Fei-FeiLi,et al.  One-Shot Learning of Object Categories , 2006 .

[12]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[13]  Petr Hurtik,et al.  Data Preprocessing Technique for Neural Networks Based on Image Represented by a Fuzzy Function , 2020, IEEE Transactions on Fuzzy Systems.

[15]  C. L. Philip Chen,et al.  A Fuzzy Deep Model Based on Fuzzy Restricted Boltzmann Machines for High-Dimensional Data Classification , 2020, IEEE Transactions on Fuzzy Systems.

[16]  D. Hubel,et al.  Receptive fields, binocular interaction and functional architecture in the cat's visual cortex , 1962, The Journal of physiology.

[17]  Martina Danková,et al.  Towards F-transform of a Higher Degree , 2009, IFSA/EUSFLAT Conf..

[18]  Jürgen Schmidhuber,et al.  Multi-column deep neural networks for image classification , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Pietro Perona,et al.  One-shot learning of object categories , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Vilém Novák,et al.  Analysis of seasonal time series using fuzzy approach , 2010, Int. J. Gen. Syst..

[21]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[22]  I. A. Weinstein,et al.  Fuzzy Logic Module of Convolutional Neural Network for Handwritten Digits Recognition , 2016 .

[23]  Stefania Tomasiello,et al.  A review on the application of fuzzy transform in data and image compression , 2019, Soft Comput..

[24]  Vladik Kreinovich,et al.  A new reconstruction from the F-transform components , 2016, Fuzzy Sets Syst..

[25]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[26]  Chao Yang,et al.  A Survey on Deep Transfer Learning , 2018, ICANN.

[27]  L. Deng,et al.  The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , 2012, IEEE Signal Processing Magazine.

[28]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[29]  Irina Perfilieva,et al.  Scale-Space Theory, F-transform Kernels and CNN Realization , 2019, IWANN.

[30]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[32]  Vlasanek Pavel,et al.  Interpolation techniques versus F-transform in application to image reconstruction , 2014, 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE).

[33]  Thomas Brox,et al.  Striving for Simplicity: The All Convolutional Net , 2014, ICLR.

[34]  Tony Lindeberg,et al.  Scale-space theory , 2001 .

[35]  Irina Perfilieva,et al.  Fuzzy transforms: Theory and applications , 2006, Fuzzy Sets Syst..

[36]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Vladik Kreinovich,et al.  Fuzzy transforms of higher order approximate derivatives: A theorem , 2011, Fuzzy Sets Syst..

[38]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[39]  A. Bayat.  Recognition of Handwritten Digits Using Optimized Adaptive Neuro-Fuzzy Inference Systems and Effective Features , 2013 .

[40]  P. Lennie.  Receptive fields , 2003, Current Biology.

[41]  Irina Perfilieva,et al.  RELATIONSHIP BETWEEN CONVOLUTIONAL NEURAL NETWORKS AND F-TRANSFORM , 2016 .

[42]  Irina Perfilieva,et al.  Interpolation techniques versus F-transform in application to image reconstruction , 2014, FUZZ-IEEE.

[43]  Manik Varma,et al.  Character Recognition in Natural Images , 2009, VISAPP.

[44]  Rifat Edizkan,et al.  Use of wavelet-based two-dimensional scaling moments and structural features in cascade neuro-fuzzy classifiers for handwritten digit recognition , 2014, Neural Computing and Applications.

[45]  Rahul Rishi,et al.  Handwritten Digit Recognition using Adaptive Neuro-Fuzzy System and Ranked Features , 2018, 2018 International Conference on Computing, Power and Communication Technologies (GUCON).

[46]  Paul Elmore,et al.  An approach to explainable deep learning using fuzzy inference , 2017, Defense + Security.

[47]  Sven Behnke,et al.  Evaluation of Pooling Operations in Convolutional Architectures for Object Recognition , 2010, ICANN.

[48]  Petr Hurtík,et al.  Differentiation by the F-transform and application to edge detection , 2016, Fuzzy Sets Syst..

[49]  Tsuyoshi Murata,et al.  {m , 1934, ACML.

[50]  Xavier Gastaldi,et al.  Shake-Shake regularization , 2017, ArXiv.

[51]  Luca Maria Gambardella,et al.  Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition , 2010, ArXiv.