Harmonic Analysis and Machine Learning

Title of dissertation: Harmonic Analysis and Machine Learning Michael Pekala Doctor of Philosophy, 2018 Dissertation directed by: Professor Wojciech Czaja and Professor Doron Levy Department of Mathematics This dissertation considers data representations that lie at the interesection of harmonic analysis and neural networks. The unifying theme of this work is the goal for robust and reliable machine learning. Our specific contributions include a new variant of scattering transforms based on a Haar-type directional wavelet, a new study of deep neural network instability in the context of remote sensing problems, and new empirical studies of biomedical applications of neural networks. Harmonic Analysis and Machine Learning

[1]  Emmanuel J. Candès,et al.  Curvelets and Curvilinear Integrals , 2001, J. Approx. Theory.

[2]  Jens Krommweh.  An Orthonormal Basis of Directional Haar Wavelets on Triangles , 2009 .

[3]  Leon A. Gatys,et al.  A Neural Algorithm of Artistic Style , 2015, ArXiv.

[4]  Chong Wang,et al.  Automatic segmentation of nine retinal layer boundaries in OCT images of non-exudative AMD patients using deep learning and graph search. , 2017, Biomedical optics express.

[5]  Robert Houska,et al.  The nonexistence of shearlet scaling functions , 2012 .

[6]  Xu Chen,et al.  Deep Haar Scattering Networks , 2015, ArXiv.

[7]  Yao Zhao,et al.  Adversarial Attacks and Defences Competition , 2018, ArXiv.

[8]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[9]  Philippe Burlina,et al.  Automatic screening of age-related macular degeneration and retinal abnormalities , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[10]  Delia Cabrera DeBuc,et al.  A Review of Algorithms for Segmentation of Retinal Image Data Using Optical Coherence Tomography , 2011 .

[11]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[12]  Seyed-Mohsen Moosavi-Dezfooli,et al.  The Robustness of Deep Networks: A Geometrical Perspective , 2017, IEEE Signal Processing Magazine.

[13]  Jean-Pierre Antoine,et al.  Image analysis with two-dimensional continuous wavelet transform , 1993, Signal Process..

[14]  David A. Wagner,et al.  Towards Evaluating the Robustness of Neural Networks , 2016, 2017 IEEE Symposium on Security and Privacy (SP).

[15]  Neil J. Joshi,et al.  Automated Grading of Age-Related Macular Degeneration From Color Fundus Images Using Deep Convolutional Neural Networks , 2017, JAMA ophthalmology.

[16]  Glenn J Jaffe,et al.  Macular morphology and visual acuity in the comparison of age-related macular degeneration treatments trials. , 2013, Ophthalmology.

[17]  N. Bressler.  Age-related macular degeneration is the leading cause of blindness... , 2004, JAMA.

[18]  Wojciech Czaja,et al.  Anisotropic shearlet transforms for L2 (Rk) , 2014 .

[19]  Mark J J P van Grinsven,et al.  Automated Staging of Age-Related Macular Degeneration Using Optical Coherence Tomography. , 2017, Investigative ophthalmology & visual science.

[20]  Jun Kong,et al.  A Hybrid Approach for Incorporating Deep Visual Features and Side Channel Information with Applications to AMD Detection , 2017, 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA).

[21]  Samy Bengio,et al.  Adversarial examples in the physical world , 2016, ICLR.

[22]  Demetrio Labate,et al.  Optimally Sparse Multidimensional Representation Using Shearlets , 2007, SIAM J. Math. Anal..

[23]  Gábor Márk Somfai,et al.  Real-Time Automatic Segmentation of Optical Coherence Tomography Volume Data of the Macular Region , 2015, PloS one.

[24]  Jean-Pierre Antoine,et al.  Two-dimensional directional wavelets and the scale-angle representation , 1996, Signal Process..

[25]  Artur Przelaskowski,et al.  Wavelet-like selective representations of multidirectional structures: a mammography case , 2018, Pattern Analysis and Applications.

[26]  David Svoboda,et al.  Algorithms for Efficient Computation of Convolution , 2013 .

[27]  John J. Benedetto,et al.  The Balian-Low theorem and regularity of Gabor systems , 2003 .

[28]  Leo Breiman,et al.  Classification and Regression Trees , 1984 .

[29]  P T de Jong,et al.  An international classification and grading system for age-related maculopathy and age-related macular degeneration , 1995 .

[30]  Ronald Klein,et al.  The prevalence of age-related eye diseases and visual impairment in aging: current estimates. , 2013, Investigative ophthalmology & visual science.

[31]  Truong Q. Nguyen,et al.  Wavelets and filter banks , 1996 .

[32]  Jeffrey D. Blanchard,et al.  Crystallographic Haar-Type Composite Dilation Wavelets , 2011 .

[33]  Maneesh Kumar Singh,et al.  Lipschitz Properties for Deep Convolutional Networks , 2017, ArXiv.

[34]  Edward Wilson,et al.  Some simple Haar-type wavelets in higher dimensions , 2007 .

[35]  Delia Cabrera DeBuc,et al.  Deep Learning based Retinal OCT Segmentation , 2018, Comput. Biol. Medicine.

[36]  Trevor Hastie,et al.  The Elements of Statistical Learning , 2001 .

[37]  Wojciech Czaja,et al.  Scattering transforms and classification of hyperspectral images , 2018, Defense + Security.

[38]  Johanna M Seddon,et al.  Prediction model for prevalence and incidence of advanced age-related macular degeneration based on genetic, demographic, and environmental variables. , 2009, Investigative ophthalmology & visual science.

[39]  Vishal M. Patel,et al.  Directional Multiscale Processing of Images Using Wavelets with Composite Dilations , 2012, Journal of Mathematical Imaging and Vision.

[40]  Jerry L Prince,et al.  Retinal layer segmentation of macular OCT images using boundary classification , 2013, Biomedical optics express.

[41]  Joakim Andén,et al.  Joint time-frequency scattering for audio classification , 2015, 2015 IEEE 25th International Workshop on Machine Learning for Signal Processing (MLSP).

[42]  M. Schwartz,et al.  The retina as a window to the brain—from eye research to CNS disorders , 2013, Nature Reviews Neurology.

[43]  Yue Wu,et al.  Deep-Learning Based, Automated Segmentation of Macular Edema in Optical Coherence Tomography , 2017, bioRxiv.

[44]  Gwénolé Quellec,et al.  Deep image mining for diabetic retinopathy screening , 2016, Medical Image Anal..

[45]  Michael J. A. Girard,et al.  A Deep Learning Approach to Digitally Stain Optical Coherence Tomography Images of the Optic Nerve Head , 2017, Investigative ophthalmology & visual science.

[46]  Bianca S. Gerendas,et al.  Supervised learning and dimension reduction techniques for quantification of retinal fluid in optical coherence tomography images , 2017, Eye.

[47]  David A. Wagner,et al.  Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples , 2018, ICML.

[48]  D. Labate,et al.  Sparse Multidimensional Representations using Anisotropic Dilation and Shear Operators , 2006 .

[49]  D. Walnut,et al.  Fundamental Papers in Wavelet Theory , 2006 .

[50]  David A. Forsyth,et al.  NO Need to Worry about Adversarial Examples in Object Detection in Autonomous Vehicles , 2017, ArXiv.

[51]  Dinesh Kumar,et al.  Validating retinal fundus image analysis algorithms: issues and a proposal. , 2013, Investigative ophthalmology & visual science.

[52]  Gordon Christie,et al.  Functional Map of the World , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[53]  Gerlind Plonka,et al.  Directional Haar Wavelet Frames on Triangles , 2009 .

[54]  Pascal A. Dufour,et al.  Graph-Based Multi-Surface Segmentation of OCT Data Using Trained Hard and Soft Constraints , 2013, IEEE Transactions on Medical Imaging.

[55]  Aaron Y. Lee,et al.  Deep learning is effective for the classification of OCT images of normal versus Age-related Macular Degeneration , 2016, bioRxiv.

[56]  Xiao Xiang Zhu,et al.  Deep Learning in Remote Sensing: A Comprehensive Review and List of Resources , 2017, IEEE Geoscience and Remote Sensing Magazine.

[57]  B. Torrésani,et al.  Wavelets: Mathematics and Applications , 1994 .

[58]  Yuan Yu,et al.  TensorFlow: A system for large-scale machine learning , 2016, OSDI.

[59]  Helmut Bölcskei,et al.  Discrete Deep Feature Extraction: A Theory and New Architectures , 2016, ICML.

[60]  Jonathon Shlens,et al.  Explaining and Harnessing Adversarial Examples , 2014, ICLR.

[61]  M. Abràmoff,et al.  Improved Automated Detection of Diabetic Retinopathy on a Publicly Available Dataset Through Integration of Deep Learning. , 2016, Investigative ophthalmology & visual science.

[62]  Chih-Jen Lin,et al.  LIBSVM: A library for support vector machines , 2011, TIST.

[63]  Nicholas Carlini,et al.  On the Robustness of the CVPR 2018 White-Box Adversarial Example Defenses , 2018, ArXiv.

[64]  Steven Salzberg,et al.  On Comparing Classifiers: Pitfalls to Avoid and a Recommended Approach , 1997, Data Mining and Knowledge Discovery.

[65]  Daniel Weinberg.  Multiscale and Directional Representations of High-Dimensional Information Content in Remotely Sensed Data , 2015 .

[66]  Sina Farsiu,et al.  Kernel regression based segmentation of optical coherence tomography images with diabetic macular edema. , 2015, Biomedical optics express.

[67]  Robert B. Gramacy,et al.  Ja n 20 08 Bayesian Treed Gaussian Process Models with an Application to Computer Modeling , 2009 .

[68]  E. Candès,et al.  Ridgelets: a key to higher-dimensional intermittency? , 1999, Philosophical Transactions of the Royal Society of London. Series A: Mathematical, Physical and Engineering Sciences.

[69]  Wojciech Czaja,et al.  Rotationally Invariant Time–Frequency Scattering Transforms , 2017, Journal of Fourier Analysis and Applications.

[70]  Wang-Q Lim,et al.  The Theory of Wavelets with Composite Dilations , 2006 .

[71]  Yiran Li,et al.  Feature extraction in image processing and deep learning , 2018 .

[72]  James M. Murphy.  Anisotropic harmonic analysis and integration of remotely sensed data , 2015 .

[73]  W. Czaja,et al.  Analysis of time-frequency scattering transforms , 2016, Applied and Computational Harmonic Analysis.

[74]  Minh N. Do,et al.  Multidimensional Filter Banks and Multiscale Geometric Representations , 2012, Found. Trends Signal Process..

[75]  Wojciech Czaja,et al.  Isotropic Shearlet Analogs for L 2(ℝ k ) and Localization Operators , 2012 .

[76]  Philippe Burlina,et al.  Automated segmentation of geographic atrophy of the retinal epithelium via random forests in AREDS color fundus images , 2015, Comput. Biol. Medicine.

[77]  Stéphane Mallat,et al.  Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination , 2013, 2013 IEEE Conference on Computer Vision and Pattern Recognition.

[78]  Stéphane Mallat,et al.  Deep roto-translation scattering for object classification , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[79]  Philippe Burlina,et al.  Detection of age-related macular degeneration via deep learning , 2016, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[80]  Joan Bruna,et al.  Intriguing properties of neural networks , 2013, ICLR.

[81]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[82]  Yoshua Bengio,et al.  The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[83]  Neil M Bressler,et al.  Vision-related function after ranibizumab treatment by better- or worse-seeing eye: clinical trial results from MARINA and ANCHOR. , 2010, Ophthalmology.

[84]  B. Manning.  Composite Multi resolution Analysis Wavelets , 2012 .

[85]  Xu Chen,et al.  Unsupervised Deep Haar Scattering on Graphs , 2014, NIPS.

[86]  S. Mallat.  A wavelet tour of signal processing , 1998 .

[87]  Robert A. Schowengerdt,et al.  Remote sensing, models, and methods for image processing , 1997 .

[88]  E. Candès,et al.  New tight frames of curvelets and optimal representations of objects with piecewise C2 singularities , 2004 .

[89]  C. Heil,et al.  Accuracy of Lattice Translates of Several Multidimensional Refinable Functions , 1998 .

[90]  Wojciech Czaja,et al.  Adversarial examples in remote sensing , 2018, SIGSPATIAL/GIS.

[91]  Wang-Q Lim,et al.  Wavelets with composite dilations and their MRA properties , 2006 .

[92]  Duen Horng Chau,et al.  ShapeShifter: Robust Physical Adversarial Attack on Faster R-CNN Object Detector , 2018, ECML/PKDD.

[93]  K. Laghari,et al.  Role of B-scan ultrasonography in pre-operative cataract patients. , 2010, International journal of health sciences.

[94]  Dawn Song,et al.  Robust Physical-World Attacks on Deep Learning Models , 2017, 1707.08945.

[95]  Kris Hauser,et al.  Incorporating side-channel information into convolutional neural networks for robotic tasks , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[96]  Seyed-Mohsen Moosavi-Dezfooli,et al.  Robustness of classifiers: from adversarial to random noise , 2016, NIPS.

[97]  Stéphane Mallat,et al.  Group Invariant Scattering , 2011, ArXiv.

[98]  David Wagner,et al.  Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods , 2017, AISec@CCS.

[99]  Gitta Kutyniok,et al.  Introduction to Shearlets , 2012 .

[100]  Peter A. Calabresi,et al.  Towards Topological Correct Segmentation of Macular OCT from Cascaded FCNs , 2017, FIFI/OMIA@MICCAI.

[101]  Weilin Li.  Topics in Harmonic Analysis, Sparse Representations, and Data Analysis , 2018 .

[102]  B. Schmauch,et al.  Deep learning approach for diabetic retinopathy screening , 2016 .

[103]  Rishab Gargeya,et al.  Automated Identification of Diabetic Retinopathy Using Deep Learning. , 2017, Ophthalmology.

[104]  Geoffrey E. Hinton,et al.  Learning to Detect Roads in High-Resolution Aerial Images , 2010, ECCV.

[105]  Wang-Q Lim,et al.  Wavelets with composite dilations , 2004 .

[106]  Philippe Burlina,et al.  Automated detection of drusen in the macula , 2009, 2009 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[107]  Luca Maria Gambardella,et al.  Deep Neural Networks Segment Neuronal Membranes in Electron Microscopy Images , 2012, NIPS.

[108]  Jeffrey D. Blanchard.  Minimally Supported Frequency Composite Dilation Wavelets , 2009 .

[109]  Steffen Schmitz-Valckenberg,et al.  Geographic atrophy: clinical features and potential therapeutic approaches. , 2014, Ophthalmology.

[110]  Philippe Burlina,et al.  Comparing humans and deep learning performance for grading AMD: A study in using universal deep features and transfer learning for automated AMD analysis , 2017, Comput. Biol. Medicine.

[111]  E. King,et al.  Wavelet and frame theory: frame bound gaps, generalized shearlets, Grassmannian fusion frames, and p-adic wavelets , 2009 .

[112]  S. Mallat,et al.  Invariant Scattering Convolution Networks , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[113]  Sergey Zagoruyko,et al.  Scaling the Scattering Transform: Deep Hybrid Networks , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[114]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[115]  Kilian Q. Weinberger,et al.  Densely Connected Convolutional Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[116]  Christopher Sansing,et al.  Gabor frames and directional time–frequency analysis , 2008 .

[117]  Ananthram Swami,et al.  The Limitations of Deep Learning in Adversarial Settings , 2015, 2016 IEEE European Symposium on Security and Privacy (EuroS&P).

[118]  Radford Juang,et al.  Automatic segmentation of the left-ventricular cavity and atrium in 3D ultrasound using graph cuts and the radial symmetry transform , 2011, 2011 IEEE International Symposium on Biomedical Imaging: From Nano to Macro.

[119]  Thomas Wiatowski,et al.  A Mathematical Theory of Deep Convolutional Neural Networks for Feature Extraction , 2015, IEEE Transactions on Information Theory.

[120]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[121]  G. Weiss,et al.  A First Course on Wavelets , 1996 .

[122]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[123]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[124]  Thomas Brox,et al.  3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation , 2016, MICCAI.

[125]  Ananthram Swami,et al.  Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples , 2016, ArXiv.

[126]  Wojciech Czaja,et al.  Composite wavelet representations for reconstruction of missing data , 2013, Defense, Security, and Sensing.

[127]  Logan Engstrom,et al.  Synthesizing Robust Adversarial Examples , 2017, ICML.

[128]  Jing Tian,et al.  Performance evaluation of automated segmentation software on optical coherence tomography volume data , 2016, Journal of biophotonics.