Computations, optimization and tuning of deep feedforward neural networks

This article presents an overview of the generalized formulations of the computations, optimization, and tuning of a deep feedforward neural network. A small network has been used to systematically explain the computing steps, which were then used to establish the generalized forms of the computations in forward and backward propagations for larger networks. Additionally, some of the commonly used cost functions, activation functions, optimization algorithms, and hyper-parameters tuning approaches have been discussed.

[1]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[2]  Mark Tommerdahl,et al.  Auditory and tactile frequency representations are co-embedded in modality-defined cortical sensory systems , 2020, NeuroImage.

[3]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[4]  Nancy Kanwisher,et al.  Toward a universal decoder of linguistic meaning from brain activation , 2018, Nature Communications.

[5]  Lexi E. Crommett,et al.  Multimodal frequency representations are embedded in modality-defined cortical sensory systems , 2019, bioRxiv.

[6]  Konrad P. Körding,et al.  Toward an Integration of Deep Learning and Neuroscience , 2016, bioRxiv.

[7]  J. Gallant,et al.  Identifying natural images from human brain activity , 2008, Nature.

[8]  Farid U. Dowla,et al.  Backpropagation Learning for Multilayer Feed-Forward Neural Networks Using the Conjugate Gradient Method , 1991, Int. J. Neural Syst..

[9]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[10]  Kyong Hwan Jin,et al.  Predicting cognitive decline with deep learning of brain metabolism and amyloid imaging , 2017, Behavioural Brain Research.

[11]  Md. Shoaibur Rahman.  Spectral remapping of natural signals , 2019 .

[12]  Marcel van Gerven,et al.  Unsupervised Feature Learning Improves Prediction of Human Brain Activity in Response to Natural Images , 2014, PLoS Comput. Biol..

[13]  Jeffrey M Yau,et al.  Somatosensory interactions reveal feature-dependent computations. , 2019, Journal of neurophysiology.

[14]  Thomas L. Griffiths,et al.  Supplementary Information for Natural Speech Reveals the Semantic Maps That Tile Human Cerebral Cortex , 2022 .

[15]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[16]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[17]  Erik Cambria,et al.  Recent Trends in Deep Learning Based Natural Language Processing , 2017, IEEE Comput. Intell. Mag..

[18]  Daniel Svozil,et al.  Introduction to multi-layer feed-forward neural networks , 1997 .

[19]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[20]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[21]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[22]  Md. Shoaibur Rahman,et al.  Selective Attention Gates the Interactive Crossmodal Coupling between Perceptual Systems , 2017, Current Biology.

[23]  Xinghuo Yu,et al.  A general backpropagation algorithm for feedforward neural networks learning , 2002, IEEE Trans. Neural Networks.

[24]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[25]  Christopher D. Manning,et al.  Advances in natural language processing , 2015, Science.

[26]  Mario Bertero,et al.  The Stability of Inverse Problems , 1980 .

[27]  Andrew Y. Ng,et al.  Parsing Natural Scenes and Natural Language with Recursive Neural Networks , 2011, ICML.

[28]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[29]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[30]  M. Asaduzzaman,et al.  Mathematical modeling of the heart , 2010, International Conference on Electrical & Computer Engineering (ICECE 2010).

[31]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[32]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  M. Rahman,et al.  Mathematical modeling of blood flow , 2012, 2012 International Conference on Informatics, Electronics & Vision (ICIEV).

[34]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[35]  Md Shoaibur Rahman.  Computational design of cardiac activity , 2011 .

[36]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[37]  F. Tong,et al.  Decoding the visual and subjective contents of the human brain , 2005, Nature Neuroscience.

[38]  Ning Qian,et al.  On the momentum term in gradient descent learning algorithms , 1999, Neural Networks.

[39]  Jian Sun,et al.  Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).