Bayesian Multi-scale Convolutional Neural Network for Motif Occupancy Identification

Convolutional neural network (CNN) has been successfully used for the identification of motif occupancy. However, the CNN architecture requires varying length instead of fixed-length filters due to different motif lengths. Moreover, plain neural networks with single point estimation for weights suffer from over-fitting, which is more likely to occur as increasing parameters for multi-scale modeling.Hence, we have designed a Bayesian Multi-scale CNN. The model employs convolutional filters of different scales to extract latent features of DNA sequence, and incorporates Bayesian architecture which regards multi-scale weights as random variables. We further stack two sequential convolutional operations for mean and variance respectively, and apply Bayes by Back prop for posterior estimation of weights. Results have shown that our method not only improved the prediction performance for motif occupancy identification, but also prevented over-fitting due to the capability of Bayesian neural network. The model has also developed a measure of uncertainty estimation for model assessment.

[1]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[2]  Ronan Sicre,et al.  Particular object retrieval with integral max-pooling of CNN activations , 2015, ICLR.

[3]  Michael I. Jordan,et al.  Mean Field Theory for Sigmoid Belief Networks , 1996, J. Artif. Intell. Res..

[4]  Kumar Shridhar,et al.  Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference , 2018 .

[5]  Wilfred W. Li,et al.  MEME: discovering and analyzing DNA and protein sequence motifs , 2006, Nucleic Acids Res..

[6]  Yoshua Bengio,et al.  Série Scientifique Scientific Series Incorporating Second-order Functional Knowledge for Better Option Pricing Incorporating Second-order Functional Knowledge for Better Option Pricing , 2022 .

[7]  Karl J. Friston,et al.  Variational free energy and the Laplace approximation , 2007, NeuroImage.

[8]  Atina G. Coté,et al.  Evaluation of methods for modeling transcription factor sequence specificity , 2013, Nature Biotechnology.

[9]  Geoffrey E. Hinton,et al.  Rectified Linear Units Improve Restricted Boltzmann Machines , 2010, ICML.

[10]  Michael I. Jordan,et al.  Bayesian parameter estimation via variational methods , 2000, Stat. Comput..

[11]  Oriol Vinyals,et al.  Bayesian Recurrent Neural Networks , 2017, ArXiv.

[12]  David K. Gifford,et al.  Convolutional neural network architectures for predicting DNA–protein binding , 2016, Bioinform..

[13]  Alex Graves,et al.  Practical Variational Inference for Neural Networks , 2011, NIPS.

[14]  A. Kiureghian,et al.  Aleatory or epistemic? Does it matter? , 2009 .

[15]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Geoffrey E. Hinton,et al.  A View of the Em Algorithm that Justifies Incremental, Sparse, and other Variants , 1998, Learning in Graphical Models.

[18]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[19]  A. Nair,et al.  A coding measure scheme employing electron-ion interaction pseudopotential (EIIP) , 2006, Bioinformation.

[20]  Hongkai Ji,et al.  Dynamic motif occupancy (DynaMO) analysis identifies transcription factors and their binding sites driving dynamic biological processes , 2017, Nucleic acids research.

[21]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[22]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[23]  Myunghee Cho Paik,et al.  Uncertainty quantification using Bayesian neural networks in classification: Application to ischemic stroke lesion segmentation , 2018 .

[24]  Vladimir B. Bajic,et al.  DeepGSR: an optimized deep-learning structure for the recognition of genomic signals and regions , 2018, Bioinform..

[25]  Y. Yao,et al.  On Early Stopping in Gradient Descent Learning , 2007 .

[26]  Morteza Mohammad Noori,et al.  Enhanced Regulatory Sequence Prediction Using Gapped k-mer Features , 2014, PLoS Comput. Biol..

[27]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[28]  Mikael Bodén,et al.  MEME Suite: tools for motif discovery and searching , 2009, Nucleic Acids Res..

[29]  Casey S Greene,et al.  Bayesian deep learning for single-cell analysis , 2018, Nature Methods.

[30]  Julien Cornebise,et al.  Weight Uncertainty in Neural Networks , 2015, ArXiv.

[31]  Swetlana Nikolajewa,et al.  DiProDB: a database for dinucleotide properties , 2008, Nucleic Acids Res..

[32]  Thomas B. Schön,et al.  Evaluating Scalable Bayesian Deep Learning Methods for Robust Computer Vision , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Mohamed A. Ismail,et al.  MProfiler: A Profile-Based Method for DNA Motif Discovery , 2009, PRIB.