Deep learning and radiomics in precision medicine

ABSTRACT Introduction: The radiological reading room is undergoing a paradigm shift to a symbiosis of computer science and radiology using artificial intelligence integrated with machine and deep learning with radiomics to better define tissue characteristics. The goal is to use integrated deep learning and radiomics with radiological parameters to produce a personalized diagnosis for a patient. Areas covered: This review provides an overview of historical and current deep learning and radiomics methods in the context of precision medicine in radiology. A literature search for ‘Deep Learning’, ‘Radiomics’, ‘Machine learning’, ‘Artificial Intelligence’, ‘Convolutional Neural Network’, ‘Generative Adversarial Network’, ‘Autoencoders’, Deep Belief Networks”, Reinforcement Learning”, and ‘Multiparametric MRI’ was performed in PubMed, ArXiv, Scopus, CVPR, SPIE, IEEE Xplore, and NIPS to identify articles of interest. Expert opinion: In conclusion, both deep learning and radiomics are two rapidly advancing technologies that will unite in the future to produce a single unified framework for clinical decision support with a potential to completely revolutionize the field of precision medicine.

[1]  Y. Tabak,et al.  An Automated Model to Identify Heart Failure Patients at Risk for 30-Day Readmission or Death Using Electronic Medical Record Data , 2010, Medical care.

[2]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[3]  Geoffrey E. Hinton,et al.  A Learning Algorithm for Boltzmann Machines , 1985, Cogn. Sci..

[4]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[5]  Benoit B. Mandelbrot,et al.  Fractal Geometry of Nature , 1984 .

[6]  Richard S. Sutton,et al.  Temporal credit assignment in reinforcement learning , 1984 .

[7]  Alex Lallement,et al.  Survey on deep learning for radiotherapy , 2018, Comput. Biol. Medicine.

[8]  W. A. Clark,et al.  Simulation of self-organizing systems by digital computer , 1954, Trans. IRE Prof. Group Inf. Theory.

[9]  Sunghwan Sohn,et al.  Mayo clinical Text Analysis and Knowledge Extraction System (cTAKES): architecture, component evaluation and applications , 2010, J. Am. Medical Informatics Assoc..

[10]  Bolei Zhou,et al.  Network Dissection: Quantifying Interpretability of Deep Visual Representations , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Robert M. Haralick,et al.  Textural Features for Image Classification , 1973, IEEE Trans. Syst. Man Cybern..

[13]  Feng Liu,et al.  Deep Learning and Its Applications in Biomedicine , 2018, Genom. Proteom. Bioinform..

[14]  Vladimir Braverman,et al.  DreamNLP: Novel NLP System for Clinical Report Metadata Extraction using Count Sketch Data Streaming Algorithm: Preliminary Results , 2018, ArXiv.

[15]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[16]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  Jason Yosinski,et al.  Deep neural networks are easily fooled: High confidence predictions for unrecognizable images , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Jürgen Schmidhuber,et al.  Reinforcement Learning with Interacting Continually Running Fully Recurrent Networks , 1990 .

[19]  Michael I. Jordan,et al.  On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes , 2001, NIPS.

[20]  Jürgen Schmidhuber,et al.  A local learning algorithm for dynamic feedforward and recurrent networks , 1990, Forschungsberichte, TU Munich.

[21]  Graham W. Taylor,et al.  Deconvolutional networks , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  Vishwa S. Parekh,et al.  Multiparametric radiomics methods for breast cancer tissue characterization using radiological imaging , 2018, Breast Cancer Research and Treatment.

[23]  Geoffrey G. Zhang,et al.  Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels , 2017, Medical physics.

[24]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[25]  Alekseĭ Grigorʹevich Ivakhnenko,et al.  CYBERNETIC PREDICTING DEVICES , 1966 .

[26]  Richard S. Sutton,et al.  Reinforcement Learning: An Introduction , 1998, IEEE Trans. Neural Networks.

[27]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[28]  Yang Gao,et al.  End-to-End Learning of Driving Models from Large-Scale Video Datasets , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[29]  Michelle Becker,et al.  Perceptrons An Introduction To Computational Geometry , 2016 .

[30]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[31]  Gaëtan Hadjeres,et al.  Deep Learning Techniques for Music Generation - A Survey , 2017, ArXiv.

[32]  H. Aerts,et al.  Applications and limitations of radiomics , 2016, Physics in medicine and biology.

[33]  Heinz Handels,et al.  Unsupervised Pathology Detection in Medical Images using Learning-based Methods , 2018, Bildverarbeitung für die Medizin.

[34]  Jayaram K. Udupa,et al.  New variants of a method of MRI scale standardization , 2000, IEEE Transactions on Medical Imaging.

[35]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  Paul Babyn,et al.  Generative Adversarial Network in Medical Imaging: A Review , 2018, Medical Image Anal..

[37]  Vishwa Parekh,et al.  Radiomics: a new application from established techniques , 2016, Expert review of precision medicine and drug development.

[38]  Andrea Vedaldi,et al.  Understanding deep image representations by inverting them , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[39]  Pascal Vincent,et al.  Visualizing Higher-Layer Features of a Deep Network , 2009 .

[40]  Dorin Comaniciu,et al.  Multi-Scale Deep Reinforcement Learning for Real-Time 3D-Landmark Detection in CT Scans , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[41]  Geoffrey G. Zhang,et al.  Voxel size and gray level normalization of CT radiomic features in lung cancer , 2018, Scientific Reports.

[42]  Suzette J. Bielinski,et al.  Use of diverse electronic medical record systems to identify genetic risk for type 2 diabetes within a genome-wide association study , 2012, J. Am. Medical Informatics Assoc..

[43]  M. Kubát.  An Introduction to Machine Learning , 2017, Springer International Publishing.

[44]  Kunihiko Fukushima,et al.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position , 1980, Biological Cybernetics.

[45]  Seyed-Mohsen Moosavi-Dezfooli,et al.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  A. M. Turing,et al.  Computing Machinery and Intelligence , 1950, The Philosophy of Artificial Intelligence.

[47]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[48]  Jürgen Schmidhuber,et al.  Recurrent networks adjusted by adaptive critics , 1990 .

[49]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[50]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[51]  Daniel L. Rubin,et al.  Deep Learning for Brain MRI Segmentation: State of the Art and Future Directions , 2017, Journal of Digital Imaging.

[52]  Bram Bakker,et al.  Reinforcement Learning with Long Short-Term Memory , 2001, NIPS.

[53]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[54]  Stefan Wermter,et al.  Continual Lifelong Learning with Neural Networks: A Review , 2019, Neural Networks.

[55]  Lawrence D. Jackel,et al.  Handwritten Digit Recognition with a Back-Propagation Network , 1989, NIPS.

[56]  Antonio Torralba,et al.  HOGgles: Visualizing Object Detection Features , 2013, 2013 IEEE International Conference on Computer Vision.

[57]  Andrew W. Moore,et al.  Reinforcement Learning: A Survey , 1996, J. Artif. Intell. Res..

[58]  Roberto Cipolla,et al.  SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Vishwa S. Parekh,et al.  Multiparametric deep learning tissue signatures for a radiological biomarker of breast cancer: Preliminary results , 2019, Medical physics.

[60]  Graham W. Taylor,et al.  Adaptive deconvolutional networks for mid and high level feature learning , 2011, 2011 International Conference on Computer Vision.

[61]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[62]  Patrick Granton,et al.  Radiomics: extracting more information from medical images using advanced feature analysis. , 2012, European journal of cancer.

[63]  A. Hall,et al.  Adaptive Switching Circuits , 2016 .

[64]  Max Welling,et al.  Auto-Encoding Variational Bayes , 2013, ICLR.

[65]  Jason Yosinski,et al.  Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks , 2016, ArXiv.

[66]  Michael A Jacobs,et al.  Integrated radiomic framework for breast cancer and tumor biology using advanced machine learning and multiparametric MRI , 2017, npj Breast Cancer.

[67]  Mehryar Mohri,et al.  AdaNet: Adaptive Structural Learning of Artificial Neural Networks , 2016, ICML.

[68]  Robert King,et al.  Textural features corresponding to textural properties , 1989, IEEE Trans. Syst. Man Cybern..

[69]  Paul Smolensky,et al.  Information processing in dynamical systems: foundations of harmony theory , 1986 .

[70]  Eugenio Culurciello,et al.  An Analysis of Deep Neural Network Models for Practical Applications , 2016, ArXiv.

[71]  A. G. Ivakhnenko,et al.  Polynomial Theory of Complex Systems , 1971, IEEE Trans. Syst. Man Cybern..

[72]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[73]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[74]  Aleksander Madry,et al.  Towards Deep Learning Models Resistant to Adversarial Attacks , 2017, ICLR.

[75]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[76]  P. Lambin,et al.  Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach , 2014, Nature Communications.

[77]  Jen-Tzung Chien,et al.  Deep reinforcement learning for automated radiation adaptation in lung cancer , 2017, Medical physics.

[78]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[79]  Vince D. Calhoun,et al.  Variational Autoencoders for Feature Detection of Magnetic Resonance Imaging Data , 2016, ArXiv.

[80]  Pascal Vincent,et al.  Stacked Denoising Autoencoders: Learning Useful Representations in a Deep Network with a Local Denoising Criterion , 2010, J. Mach. Learn. Res..

[81]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[82]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[83]  Luca Maria Gambardella,et al.  Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence Flexible, High Performance Convolutional Neural Networks for Image Classification , 2022 .

[84]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[85]  Andre Dekker,et al.  Radiomics: the process and the challenges. , 2012, Magnetic resonance imaging.

[86]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[87]  Martín Abadi,et al.  Adversarial Patch , 2017, ArXiv.

[88]  Vishwa S. Parekh,et al.  Radiomic Synthesis Using Deep Convolutional Neural Networks , 2018, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[89]  Arvind Satyanarayan,et al.  The Building Blocks of Interpretability , 2018 .

[90]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[91]  Chenchen Liu,et al.  How convolutional neural networks see the world - A survey of convolutional neural network visualization methods , 2018, Math. Found. Comput..

[92]  Leon A. Gatys,et al.  Image Style Transfer Using Convolutional Neural Networks , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[93]  Dwarikanath Mahapatra,et al.  Semi-supervised Segmentation of Optic Cup in Retinal Fundus Images Using Variational Autoencoder , 2017, MICCAI.

[94]  Yee Whye Teh,et al.  A Fast Learning Algorithm for Deep Belief Nets , 2006, Neural Computation.

[95]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[96]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[97]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[98]  Dimitris N. Metaxas,et al.  StackGAN: Text to Photo-Realistic Image Synthesis with Stacked Generative Adversarial Networks , 2016, 2017 IEEE International Conference on Computer Vision (ICCV).

[99]  Kenneth I. Laws,et al.  Rapid Texture Identification , 1980, Optics & Photonics.

[100]  Kouichi Sakurai,et al.  One Pixel Attack for Fooling Deep Neural Networks , 2017, IEEE Transactions on Evolutionary Computation.

[101]  Jürgen Schmidhuber,et al.  Reinforcement Learning in Markovian and Non-Markovian Environments , 1990, NIPS.

[102]  Thomas Brox,et al.  Inverting Visual Representations with Convolutional Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[103]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[104]  Chuan Li,et al.  Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks , 2016, ECCV.

[105]  W S McCulloch,et al.  A logical calculus of the ideas immanent in nervous activity , 1990, The Philosophy of Artificial Intelligence.

[106]  Mary M. Galloway,et al.  Texture analysis using gray level run lengths , 1974 .

[107]  Kuldip K. Paliwal,et al.  Bidirectional recurrent neural networks , 1997, IEEE Trans. Signal Process..

[108]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[109]  Demis Hassabis,et al.  Mastering the game of Go with deep neural networks and tree search , 2016, Nature.