Numerical Algorithms - Methods for Computer Vision, Machine Learning, and Graphics

Numerical Algorithms: Methods for Computer Vision, Machine Learning, and Graphics presents a new approach to numerical analysis for modern computer scientists. Using examples from a broad base of computational tasks, including data processing, computational photography, and animation, the textbook introduces numerical modeling and algorithmic design from a practical standpoint and provides insight into the theoretical tools needed to support these skills. The book covers a wide range of topicsfrom numerical linear algebra to optimization and differential equationsfocusing on real-world motivation and unifying themes. It incorporates cases from computer science research and practice, accompanied by highlights from in-depth literature on each subtopic. Comprehensive end-of-chapter exercises encourage critical thinking and build students intuition while introducing extensions of the basic material. The text is designed for advanced undergraduate and beginning graduate students in computer science and related fields with experience in calculus and linear algebra. For students with a background in discrete mathematics, the book includes some reminders of relevant continuous mathematical background.

[1]  L. R. Scott,et al.  The Mathematical Theory of Finite Element Methods , 1994 .

[2]  Yurii Nesterov,et al.  Introductory Lectures on Convex Optimization - A Basic Course , 2014, Applied Optimization.

[3]  Radford M. Neal.  Pattern Recognition and Machine Learning , 2007, Technometrics.

[4]  Andrew V. Knyazev,et al.  Toward the Optimal Preconditioned Eigensolver: Locally Optimal Block Preconditioned Conjugate Gradient Method , 2001, SIAM J. Sci. Comput..

[5]  Stig Larsson,et al.  Partial differential equations with numerical methods , 2003, Texts in applied mathematics.

[6]  Russell C. Eberhart,et al.  A new optimizer using particle swarm theory , 1995, MHS'95. Proceedings of the Sixth International Symposium on Micro Machine and Human Science.

[7]  W. T. Tutte.  How to Draw a Graph , 1963 .

[8]  Lloyd N. Trefethen,et al.  Barycentric Lagrange Interpolation , 2004, SIAM Rev..

[9]  Kaare Brandt Petersen,et al.  The Matrix Cookbook , 2006 .

[10]  Mark de Berg,et al.  Computational geometry: algorithms and applications , 1997 .

[11]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[12]  Ya-Xiang Yuan,et al.  A Nonlinear Conjugate Gradient Method with a Strong Global Convergence Property , 1999, SIAM J. Optim..

[13]  Alexander M. Bronstein,et al.  Numerical Geometry of Non-Rigid Shapes , 2009, Monographs in Computer Science.

[14]  Yousef Saad,et al.  Iterative methods for sparse linear systems , 2003 .

[15]  Jitse Niesen,et al.  Algorithm 919: A Krylov Subspace Algorithm for Evaluating the ϕ-Functions Appearing in Exponential Integrators , 2009, TOMS.

[16]  David F. Gleich,et al.  Tall and skinny QR factorizations in MapReduce architectures , 2011, MapReduce '11.

[17]  Michael M. Kazhdan,et al.  Interactive and anisotropic geometry processing using the screened Poisson equation , 2011, ACM Trans. Graph..

[18]  Roberto Manduchi,et al.  Bilateral filtering for gray and color images , 1998, Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271).

[19]  Dorothea Heiss-Czedik,et al.  An Introduction to Genetic Algorithms. , 1997, Artificial Life.

[20]  Geoffrey E. Hinton.  Training Products of Experts by Minimizing Contrastive Divergence , 2002, Neural Computation.

[21]  E. Grinspun.  Discrete differential geometry : An applied introduction , 2008, SIGGRAPH 2008.

[22]  S. Wang,et al.  Decomposition Method with a Variable Parameter for a Class of Monotone Variational Inequality Problems , 2001 .

[23]  Manolis I. A. Lourakis,et al.  Estimating the Jacobian of the Singular Value Decomposition: Theory and Applications , 2000, ECCV.

[24]  Naum Zuselevich Shor,et al.  Minimization Methods for Non-Differentiable Functions , 1985, Springer Series in Computational Mathematics.

[25]  M. Turk,et al.  Eigenfaces for Recognition , 1991, Journal of Cognitive Neuroscience.

[26]  Doug L. James,et al.  Skinning mesh animations , 2005, ACM Trans. Graph..

[27]  James T. Kajiya,et al.  The rendering equation , 1998 .

[28]  Takeo Kanade,et al.  Robust L/sub 1/ norm factorization in the presence of outliers and missing data by alternative convex programming , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[29]  Michael Elad,et al.  Sparse and Redundant Representations - From Theory to Applications in Signal and Image Processing , 2010 .

[30]  William G. Poole,et al.  A geometric theory for the QR, LU and power iterations. , 1973 .

[31]  D. Anderson,et al.  Algorithms for minimization without derivatives , 1974 .

[32]  Christian P. Robert,et al.  Monte Carlo Statistical Methods , 2005, Springer Texts in Statistics.

[33]  C. W. Clenshaw,et al.  A method for numerical integration on an automatic computer , 1960 .

[34]  Charles W. Groetsch,et al.  Lanczos' Generalized Derivative , 1998 .

[35]  Heinz H. Bauschke,et al.  What is... a Fenchel Conjugate , 2012 .

[36]  W. Cheney,et al.  Proximity maps for convex sets , 1959 .

[37]  M. Géradin,et al.  Mechanical Vibrations: Theory and Application to Structural Dynamics , 1994 .

[38]  R. Courant,et al.  Über die partiellen Differenzengleichungen der mathematischen Physik , 1928 .

[39]  I. Daubechies,et al.  Iteratively reweighted least squares minimization for sparse recovery , 2008, 0807.0575.

[40]  Martial Hebert,et al.  Smoothing-based Optimization , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Alston S. Householder,et al.  Unitary Triangularization of a Nonsymmetric Matrix , 1958, JACM.

[42]  Michael Frankfurter,et al.  Numerical Recipes In C The Art Of Scientific Computing , 2016 .

[43]  Mirela Ben-Chen,et al.  Complex Barycentric Coordinates with Applications to Planar Shape Deformation , 2009, Comput. Graph. Forum.

[44]  W. Givens.  Computation of Plain Unitary Rotations Transforming a General Matrix to Triangular Form , 1958 .

[45]  Michael A. Saunders,et al.  LSQR: An Algorithm for Sparse Linear Equations and Sparse Least Squares , 1982, TOMS.

[46]  R. Tibshirani.  Regression Shrinkage and Selection via the Lasso , 1996 .

[47]  Matthias Hein,et al.  Sparse recovery by thresholded non-negative least squares , 2011, NIPS.

[48]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[49]  Kenneth Levenberg.  A METHOD FOR THE SOLUTION OF CERTAIN NON – LINEAR PROBLEMS IN LEAST SQUARES , 1944 .

[50]  Alexander G. Ramm,et al.  A scheme for stable numerical differentiation , 2006 .

[51]  Andrew P. Witkin,et al.  Untangling cloth , 2003, ACM Trans. Graph..

[52]  Sheldon Axler,et al.  Down With Determinants ! , 2002 .

[53]  Sébastien Bubeck,et al.  Theory of Convex Optimization for Machine Learning , 2014, ArXiv.

[54]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[55]  Marc Teboulle,et al.  A Fast Iterative Shrinkage-Thresholding Algorithm for Linear Inverse Problems , 2009, SIAM J. Imaging Sci..

[56]  Daniela Fischer.  Differential Equations Dynamical Systems And An Introduction To Chaos , 2016 .

[57]  Danny C. Sorensen,et al.  Deflation Techniques for an Implicitly Restarted Arnoldi Iteration , 1996, SIAM J. Matrix Anal. Appl..

[58]  P. Sonneveld.  CGS, A Fast Lanczos-Type Solver for Nonsymmetric Linear systems , 1989 .

[59]  Michael A. Saunders,et al.  LSMR: An Iterative Algorithm for Sparse Least-Squares Problems , 2010, SIAM J. Sci. Comput..

[60]  James Kennedy,et al.  Particle swarm optimization , 2002, Proceedings of ICNN'95 - International Conference on Neural Networks.

[61]  K. Kiwiel.  Methods of Descent for Nondifferentiable Optimization , 1985 .

[62]  Stephen P. Boyd,et al.  Graph Implementations for Nonsmooth Convex Programs , 2008, Recent Advances in Learning and Control.

[63]  Michael T. Heath,et al.  Scientific Computing: An Introductory Survey , 1996 .

[64]  Mei Han An,et al.  accuracy and stability of numerical algorithms , 1991 .

[65]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[66]  Marc Alexa,et al.  As-rigid-as-possible surface modeling , 2007, Symposium on Geometry Processing.

[67]  Kilian Q. Weinberger,et al.  Unsupervised Learning of Image Manifolds by Semidefinite Programming , 2004, CVPR.

[68]  R. Fletcher.  Conjugate gradient methods for indefinite systems , 1976 .

[69]  James M. Ortega,et al.  The LLT and QR methods for symmetric tridiagonal matrices , 1963, Comput. J..

[70]  C. M. Reeves,et al.  Function minimization by conjugate gradients , 1964, Comput. J..

[71]  E. Hairer,et al.  Solving Ordinary Differential Equations II: Stiff and Differential-Algebraic Problems , 2010 .

[72]  Tom Duff,et al.  Matrix animation and polar decomposition , 1992 .

[73]  Heinz H. Bauschke,et al.  On Projection Algorithms for Solving Convex Feasibility Problems , 1996, SIAM Rev..

[74]  H. Uzawa,et al.  Preference, production, and capital: Iterative methods for concave programming , 1989 .

[75]  Jernej Barbic,et al.  Real-time large-deformation substructuring , 2011, ACM Trans. Graph..

[76]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[77]  W. Hackbusch.  Iterative Solution of Large Sparse Systems of Equations , 1993 .

[78]  Joel A. Tropp,et al.  Column subset selection, matrix factorization, and eigenvalue optimization , 2008, SODA.

[79]  D. Shepard.  A two-dimensional interpolation function for irregularly-spaced data , 1968, ACM National Conference.

[80]  Nancy S. Pollard,et al.  Real-time gradient-domain painting , 2008, ACM Trans. Graph..

[81]  Marco Dorigo,et al.  Distributed Optimization by Ant Colonies , 1992 .

[82]  Stephen P. Boyd,et al.  Distributed Optimization and Statistical Learning via the Alternating Direction Method of Multipliers , 2011, Found. Trends Mach. Learn..

[83]  J. Stoer,et al.  Introduction to Numerical Analysis , 2002 .

[84]  J. Miller.  Numerical Analysis , 1966, Nature.

[85]  F. John.  The ultrahyperbolic differential equation with four independent variables , 1938 .

[86]  Stephen P. Boyd,et al.  Applications of second-order cone programming , 1998 .

[87]  Dorin Comaniciu,et al.  Mean Shift: A Robust Approach Toward Feature Space Analysis , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[88]  Leonidas J. Guibas,et al.  Epsilon geometry: building robust algorithms from imprecise computations , 1989, SCG '89.

[89]  Martin Jaggi,et al.  Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization , 2013, ICML.

[90]  Marcin Novotni,et al.  Gomputing geodesic distances on triangular meshes , 2002 .

[91]  E. Fehlberg,et al.  Low-order classical Runge-Kutta formulas with stepsize control and their application to some heat transfer problems , 1969 .

[92]  J. Dongarra,et al.  Generalized QR factorization and its applications , 1992 .

[93]  A. Knyazev.  A Preconditioned Conjugate Gradient Method for Eigenvalue Problems and its Implementation in a Subspace , 1991 .

[94]  C. Lanczos.  Applied Analysis , 1961 .

[95]  Henk A. van der Vorst,et al.  Bi-CGSTAB: A Fast and Smoothly Converging Variant of Bi-CG for the Solution of Nonsymmetric Linear Systems , 1992, SIAM J. Sci. Comput..

[96]  D. Marquardt.  An Algorithm for Least-Squares Estimation of Nonlinear Parameters , 1963 .

[97]  William Kahan,et al.  Pracniques: further remarks on reducing truncation errors , 1965, CACM.

[98]  R. D. Richtmyer,et al.  Survey of the stability of linear finite difference equations , 1956 .

[99]  Gene H. Golub,et al.  Matrix computations , 1983 .

[100]  L. Trefethen,et al.  Stiffness of ODEs , 1993 .

[101]  Shai Shalev-Shwartz,et al.  Online Learning and Online Convex Optimization , 2012, Found. Trends Mach. Learn..

[102]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[103]  Stephen J. Wright,et al.  Numerical Optimization , 2018, Fundamental Statistical Inference.

[104]  L. Dixon,et al.  Automatic differentiation of algorithms , 2000 .

[105]  Thomas Gerstner,et al.  Numerical integration using sparse grids , 2004, Numerical Algorithms.

[106]  Jitendra Malik,et al.  Normalized cuts and image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[107]  Jack Bresenham,et al.  Algorithm for computer control of a digital plotter , 1965, IBM Syst. J..

[108]  M. Saunders,et al.  Solution of Sparse Indefinite Systems of Linear Equations , 1975 .

[109]  R. Rockafellar.  Monotone Operators and the Proximal Point Algorithm , 1976 .

[110]  Eitan Grinspun,et al.  Discrete laplace operators: no free lunch , 2007, Symposium on Geometry Processing.

[111]  N. Higham.  Computing the polar decomposition with applications , 1986 .

[112]  Joost van de Weijer,et al.  Local mode filtering , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[113]  S. Dumais.  Latent Semantic Analysis. , 2005 .

[114]  R. Freund,et al.  QMR: a quasi-minimal residual method for non-Hermitian linear systems , 1991 .

[115]  Richard Barrett,et al.  Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods , 1994, Other Titles in Applied Mathematics.

[116]  Sergei Vassilvitskii,et al.  k-means++: the advantages of careful seeding , 2007, SODA '07.

[117]  E. Polak,et al.  Note sur la convergence de méthodes de directions conjuguées , 1969 .

[118]  Philip Wolfe,et al.  An algorithm for quadratic programming , 1956 .

[119]  David G. Luenberger,et al.  Linear and nonlinear programming , 1984 .

[120]  Timothy A. Davis,et al.  Direct methods for sparse linear systems , 2006, Fundamentals of algorithms.