Sampling-based algorithms for dimension reduction

Can one compute a low-dimensional representation of any given data by looking only at its small sample, chosen cleverly on the fly? Motivated by the above question, we consider the problem of low-rank matrix approximation: given a matrix A E Rm", one wants to compute a rank-k matrix (where k << min{m, n}) nearest to A in the Frobenius norm (also known as the Hilbert-Schmidt norm). We prove that using a sample of roughly O(k/E) rows of A one can compute, with high probability, a (1 + E)-approximation to the nearest rank-k matrix. This gives an algorithm for low-rank approximation with an improved error guarantee (compared to the additive c IIA112 guarantee known earlier from the work of Frieze, Kannan, and Vempala) and running time O(Mk/l), where M is the number of non-zero entries of A. The proof is based on two sampling techniques called adaptive sampling and volume sampling, and some linear algebraic tools. Low-rank matrix approximation under the Frobenius norm is equivalent to the problem of finding a low-dimensional subspace that minimizes the sum of squared distances to given points. The general subspace approximation problem asks one to find a low-dimensional subspace that minimizes the sum of p-th powers of distances (for p > 1) to given points. We generalize our sampling techniques and prove similar sampling-based dimension reduction results for subspace approximation. However, the proof is geometric. Thesis co-supervisor: Santosh S. Vempala Title: Associate Professor of Applied Mathematics Thesis co-supervisor: Daniel A. Spielman Title: Professor of Computer Science, Yale Unversity To dear aai and baba, Acknowledgments I am greatly indebted to my advisors Daniel Spielman and Santosh Vempala for their guidance, encouragement, and patience during all these years. Both of them showed tremendous faith in me even when my progress was slow. In Spring'05, Santosh taught a wonderful course on Spectral Algorithms and Representations; most of my thesis has germinated from some questions that he posed in the class. I thank my collaborators Luis Rademacher, Grant Wang, and Kasturi Varadarajan, whose contributions have been pivotal in my research. Luis has been more than just a collaborator. I thank him for all the conversations, lunches, dinners, squash, sailing, and travel that we enjoyed together. I thank Piotr Indyk, Daniel Kleitman, and Peter Shor for agreeing to be on my thesis committee. (Finally, it was Peter Shor who attended my thesis defense and signed the thesis approval form.) My five years at MIT, including a semester spent at Georgia Tech, were made memorable by people around me: teachers, colleagues, friends. I thank my professors Daniel Kleitman, Madhu Sudan, Michel Goemans, and Rom Pinchasi for the wonderful courses they taught. I thank Subhash Khot and Prahladh Harsha for playing the roles of my mentors from time to time. I thank my roommate Shashi, my occasional cooking and chit-chat buddies Ajay, Pavithra, Kripa, Sreekar, Punya, Shubhangi, ... my friends from CMI Tejaswi, Krishna, Baskar, Debajyoti, Sourav, ... my friends from the skit group Vikram, Raghavendran, Pranava, ... and many others who made my everyday life enjoyable. Words are insufficient to express my gratitude towards my music teacher Warren Senders, who introduced me to the hell of frustration as well as the heaven of enlightenment called Hindustani music! I thank Jaikumar Radhakrishnan, Ramesh Hariharan, Narayan Kumar, K. V. Subrahmanyam, and Meena Mahajan, my professors during undergrad, for my upbringing in Theory. I thank my teachers from school and math olympiad camps for cultivating my interest in mathematics. I thank my relatives cousins, aunts, uncles and friends for asking me every year, "How many more?" and providing the external force required by Newton's first law of graduation! Above all, I thank my grandparents and my extended family, who instilled in me the liking for an academic career. I thank my parents for always giving me the freedom to choose my own path. I couldn't have reached this far if they hadn't walked me through my very early steps. This thesis is dedicated to them.