On Orthogonal Projections for Dimension Reduction and Applications in Augmented Target Loss Functions for Learning Problems

The use of orthogonal projections on high-dimensional input and target data in learning frameworks is studied. First, we investigate the relations between two standard objectives in dimension reduction, preservation of variance and of pairwise relative distances. Investigations of their asymptotic correlation as well as numerical experiments show that a projection does usually not satisfy both objectives at once. In a standard classification problem, we determine projections on the input data that balance the objectives and compare subsequent results. Next, we extend our application of orthogonal projections to deep learning tasks and introduce a general framework of augmented target loss functions. These loss functions integrate additional information via transformations and projections of the target data. In two supervised learning problems, clinical image segmentation and music information classification, the application of our proposed augmented target loss functions increases the accuracy.

[1]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[2]  Michael W. Mahoney Randomized Algorithms for Matrices and Data , 2011, Found. Trends Mach. Learn..

[3]  R. DeVore,et al.  A Simple Proof of the Restricted Isometry Property for Random Matrices , 2008 .

[4]  David Zhang,et al.  PCA-Based Spatially Adaptive Denoising of CFA Images for Single-Sensor Digital Cameras , 2009, IEEE Transactions on Image Processing.

[5]  Nicolai Meinshausen,et al.  Random Projections for Large-Scale Regression , 2017, 1701.05325.

[6]  Amir Sadeghipour,et al.  OCT biomarkers predictive for visual acuity in patients with diabetic macular edema , 2017 .

[7]  Chinmay Hegde,et al.  NuMax: A Convex Approach for Learning Near-Isometric Linear Embeddings , 2015, IEEE Transactions on Signal Processing.

[8]  Alejandro F. Frangi,et al.  Muliscale Vessel Enhancement Filtering , 1998, MICCAI.

[9]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[10]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[11]  Wang-Q Lim,et al.  ShearLab 3D , 2014, 1402.5670.

[12]  Konstantinos Kamnitsas,et al.  Anatomically Constrained Neural Networks (ACNNs): Application to Cardiac Image Enhancement and Segmentation , 2017, IEEE Transactions on Medical Imaging.

[13]  Arthur Flexer,et al.  Basic filters for convolutional neural networks applied to music: Training or design? , 2017, Neural Computing and Applications.

[14]  Rachel Ward,et al.  New and Improved Johnson-Lindenstrauss Embeddings via the Restricted Isometry Property , 2010, SIAM J. Math. Anal..

[15]  Manuel Gräf,et al.  Points on manifolds with asymptotically optimal covering radius , 2016, J. Complex..

[16]  Richard G. Baraniuk,et al.  Random Projections of Signal Manifolds , 2006, 2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings.

[17]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[18]  B. Moore Principal component analysis in linear systems: Controllability, observability, and model reduction , 1981 .

[19]  Emmanuel J. Candès,et al.  Decoding by linear programming , 2005, IEEE Transactions on Information Theory.

[20]  Dimitris Achlioptas,et al.  Database-friendly random projections: Johnson-Lindenstrauss with binary coins , 2003, J. Comput. Syst. Sci..

[21]  Roman Vershynin,et al.  Introduction to the non-asymptotic analysis of random matrices , 2010, Compressed Sensing.

[22]  Sanjoy Dasgupta,et al.  An elementary proof of a theorem of Johnson and Lindenstrauss , 2003, Random Struct. Algorithms.

[23]  Rich Caruana,et al.  Multitask Learning , 1997, Machine-mediated learning.

[24]  Gabriele Steidl,et al.  On the Robust PCA and Weiszfeld’s Algorithm , 2019, Applied Mathematics & Optimization.

[25]  Joaquim Ortega-Cerdà,et al.  Asymptotically optimal designs on compact algebraic manifolds , 2016, 1612.06729.

[26]  P. Seymour,et al.  Averaging sets: A generalization of mean values and spherical designs , 1984 .

[27]  Xavier Serra,et al.  A real-time system for measuring sound goodness in instrumental sounds , 2015 .

[28]  Jan Sijbers,et al.  Denoising of diffusion MRI using random matrix theory , 2016, NeuroImage.

[29]  Thomas Grill,et al.  Inside the spectrogram: Convolutional Neural Networks in audio processing , 2017, 2017 International Conference on Sampling Theory and Applications (SampTA).

[30]  Mert R. Sabuncu,et al.  Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels , 2018, NeurIPS.

[31]  Christine Bachoc,et al.  Tight p-fusion frames , 2012, ArXiv.

[32]  Helmut Bölcskei,et al.  Dimensionality-reduced subspace clustering , 2015, ArXiv.

[33]  J. Matousek,et al.  On variants of the Johnson–Lindenstrauss lemma , 2008 .

[34]  Richard G. Baraniuk,et al.  Random Projections of Smooth Manifolds , 2009, Found. Comput. Math..

[35]  Thomas Brox,et al.  Generating Images with Perceptual Similarity Metrics based on Deep Networks , 2016, NIPS.

[36]  Anna Breger,et al.  Quasi Monte Carlo Integration and Kernel-Based Function Approximation on Grassmannians , 2016, 1605.09165.

[37]  Heikki Mannila,et al.  Random projection in dimensionality reduction: applications to image and text data , 2001, KDD '01.

[38]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[39]  K. Ball An Elementary Introduction to Modern Convex Geometry , 1997 .

[40]  P. Harpe,et al.  Cubature Formulas, Geometrical Designs, Reproducing Kernels, and Markov Operators , 2005, math/0502312.

[41]  Stephen P. Boyd,et al.  Generalized Low Rank Models , 2014, Found. Trends Mach. Learn..

[42]  Curt Sachs,et al.  Classification of Musical Instruments: Translated from the Original German by Anthony Baines and Klaus P. Wachsmann , 1961 .

[43]  R. Kirk Steinhorst,et al.  A COMPARISON OF PRINCIPAL COMPONENTS FROM REAL AND RANDOM DATA , 1985 .

[44]  Stefano Ermon,et al.  Label-Free Supervision of Neural Networks with Physics and Domain Knowledge , 2016, AAAI.

[45]  Jing Wu,et al.  A novel benchmark model for intelligent annotation of spectral-domain optical coherence tomography scans using the example of cyst annotation , 2016, Comput. Methods Programs Biomed..

[46]  Rich Caruana,et al.  Multitask Learning , 1997, Machine Learning.

[47]  Bianca S. Gerendas,et al.  Supervised learning and dimension reduction techniques for quantification of retinal fluid in optical coherence tomography images , 2017, Eye.

[48]  Bernhard G. Bodmann,et al.  From Low- to High-Dimensional Moments Without Magic , 2016, 1601.07401.

[49]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Hugues Benoit-Cattin,et al.  Semi-supervised Learning for Segmentation Under Semantic Constraint , 2018, MICCAI.