Quantization with Multiple Constraints

Acknowledgments I first wish to express my gratitude to all the researchers and professors I met during those PhD years and who helped me with my work, either by inviting me in their lab, giving me information on their own research, comments on my papers, or simply friendly encouragements. Among them are Eve Riskin, also thank my conference partner Marcel Wagner. I warmly thank Dietmar Saupe for his help during my first year as a researcher and for accepting being part of my jury. Thanks also go to the other jury members Guy Louchard, Guy Latouche and Francis Grenez. The Computer Science Department is full of great people and I am greatly indebted to my advisor Yves Roggeman, who never failed to provide me with the material support for my work. Special notes of thanks also go to Nicole and Pascaline for their kindness and patience, and to all my colleagues. I thank the smashing trio Rod, Max and Jef, the terrific mathematician and friend Samuel, my incredibly cool officemate Steve, my stunning old-years pals Fred, Alex and Erik, the awesome Thierry and the sensational Olivier. I thank Isabelle, my parents, JP and SBA for everything. 4 Introduction This introductive chapter aims at providing a bird's eye view on the contributions of this thesis, and gives some insight on the context in which the results take place and how they can be used in practical applications. What is now called Information Theory was founded by Claude Elwood Shan-non in 1948, when his paper " A Mathematical Theory of Communication " was published. While the initial, rather practical goal of his work was to optimize the transmission of information over telegraph lines, it eventually lead to scientific breakthroughs, not only in the engineering community, but also in statistics, psychology and human sciences – among others. Shannon's communication paradigm is illustrated on Fig. 1. A entity called the transmitter communicates with a receiver via a physical medium called the channel. The channel is perturbed by noise, so that the states at the transmitter side are not the same as those read by the receiver. The sequence of physical channel states conveys some information over time. Information Theory answers the following two questions: 1. How can we measure the quantity of information sent ? 2. How much information is a channel able to transmit ? The answers, the entropy and the channel capacity, …

[1]  Kenneth Rose,et al.  Multiple description quantization by deterministic annealing , 2003, IEEE Trans. Inf. Theory.

[2]  Mort Naraghi-Pour,et al.  DPCM encoding of regenerative composite processes , 1994, IEEE Trans. Inf. Theory.

[3]  Edward M. Reingold,et al.  Index assignment for multichannel communication under failure , 2000, IEEE Trans. Inf. Theory.

[4]  Nick Roussopoulos,et al.  Nearest neighbor queries , 1995, SIGMOD '95.

[5]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[6]  Gary J. Sullivan,et al.  Recursive optimal pruning with applications to tree structured vector quantizers , 1992, IEEE Trans. Image Process..

[7]  Richard E. Ladner,et al.  Approximately optimal assignment for unequal loss protection , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[8]  David L. Neuhoff,et al.  Quantization , 2022, IEEE Trans. Inf. Theory.

[9]  W. Fischer,et al.  Sphere Packings, Lattices and Groups , 1990 .

[10]  J. Cardinal.  A Lagrangian optimization approach to complexity-constrained TSVQ , 2000, IEEE Signal Processing Letters.

[11]  Jean Cardinal,et al.  Complexity-constrained tree-structured vector quantizers , 2002, Signal Process..

[12]  Jaroslaw Domaszewicz,et al.  Design of entropy-constrained multiple-description scalar quantizers , 1994, IEEE Trans. Inf. Theory.

[13]  Michael Randolph Garey,et al.  The complexity of the generalized Lloyd - Max problem , 1982, IEEE Trans. Inf. Theory.

[14]  J. Makhoul,et al.  Vector quantization in speech coding , 1985, Proceedings of the IEEE.

[15]  Kannan Ramchandran,et al.  Efficient terrain data representation for 3D rendering using the generalized BFOS algorithm , 1997, Proceedings of International Conference on Image Processing.

[16]  Jean Cardinal.  Tree-based search for ECVQ , 2000, Proceedings DCC 2000. Data Compression Conference.

[17]  L. Ozarow,et al.  On a source-coding problem with two channels and three receivers , 1980, The Bell System Technical Journal.

[18]  Antonio Ortega,et al.  Distortion/decoding time tradeoffs in software DCT-based image coding , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[19]  Philip A. Chou,et al.  Optimal pruning with applications to tree-structured source coding and modeling , 1989, IEEE Trans. Inf. Theory.

[20]  R.W. Schafer,et al.  Digital representations of speech signals , 1975, Proceedings of the IEEE.

[21]  Antonio Ortega,et al.  Rate-complexity-distortion optimization for quadtree-based DCT coding , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[22]  C.-H. Lee,et al.  Fast closest codeword search algorithm for vector quantization , 1994 .

[23]  Robert M. Gray,et al.  An Improvement of the Minimum Distortion Encoding Algorithm for Vector Quantization , 1985, IEEE Trans. Commun..

[24]  David A. Cohn,et al.  Theory and Practice of Vector Quantizers Trained on Small Training Sets , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[25]  Antonin Guttman,et al.  R-trees: a dynamic index structure for spatial searching , 1984, SIGMOD '84.

[26]  Robert M. Gray,et al.  Speech coding based upon vector quantization , 1980, ICASSP.

[27]  Eve A. Riskin,et al.  Optimal bit allocation and best-basis selection for wavelet packets and TSVQ , 1999, IEEE Trans. Image Process..

[28]  L. H. Harper.  On an Isoperimetric Problem for Hamming Graphs , 1999, Discret. Appl. Math..

[29]  Frédérique E. Oggier,et al.  Semidefinite programs for the design of codes for delay-constrained communication in networks , 2001, Proceedings DCC 2001. Data Compression Conference.

[30]  Steven McCanne,et al.  Dynamic memory model based optimization of scalar and vector quantizer for fast image encoding , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[31]  Naoki Katoh,et al.  Resource Allocation Problems , 1998 .

[32]  Ulug Bayazit,et al.  Variable-length constrained-storage tree-structured vector quantization , 1999, IEEE Trans. Image Process..

[33]  Michelle Effros,et al.  The rate distortion region for the multiple description problem , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[34]  Vivek K. Goyal,et al.  Generalized multiple description coding with correlating transforms , 2001, IEEE Trans. Inf. Theory.

[35]  Sunil Arya,et al.  An optimal algorithm for approximate nearest neighbor searching fixed dimensions , 1998, JACM.

[36]  Richard L. Frost,et al.  Complexity-constrained trellis quantizers , 1997, IEEE Trans. Inf. Theory.

[37]  Kenneth Rose,et al.  Entropy-constrained tree-structured vector quantizer design , 1996, IEEE Trans. Image Process..

[38]  Eve A. Riskin,et al.  Lookahead in growing tree-structured vector quantizers , 1991, [Proceedings] ICASSP 91: 1991 International Conference on Acoustics, Speech, and Signal Processing.

[39]  R.M. Gray,et al.  A greedy tree growing algorithm for the design of variable rate vector quantizers [image compression] , 1991, IEEE Trans. Signal Process..

[40]  N. J. A. Sloane,et al.  Design of asymmetric multiple description lattice vector quantizers , 2000, Proceedings DCC 2000. Data Compression Conference.

[41]  Toshihide Ibaraki,et al.  Resource allocation problems - algorithmic approaches , 1988, MIT Press series in the foundations of computing.

[42]  Chang-Hsing Lee,et al.  High-speed closest codeword search algorithms for vector quantization , 1995, Signal Process..

[43]  Robert M. Gray,et al.  Global convergence and empirical consistency of the generalized Lloyd algorithm , 1986, IEEE Trans. Inf. Theory.

[44]  Sunil Arya,et al.  Algorithms for fast vector quantization , 1993, [Proceedings] DCC `93: Data Compression Conference.

[45]  Tamás Linder.  On the training distortion of vector quantizers , 2000, IEEE Trans. Inf. Theory.

[46]  Abbas El Gamal,et al.  Achievable rates for multiple descriptions , 1982, IEEE Trans. Inf. Theory.

[47]  Dennis Hui,et al.  On the encoding complexity of scalar quantizers , 1995, Proceedings of 1995 IEEE International Symposium on Information Theory.

[48]  Christian Böhm,et al.  A cost model for nearest neighbor search in high-dimensional data space , 1997, PODS.

[49]  Robert M. Gray,et al.  On Zador's entropy-constrained quantization theorem , 2001, Proceedings DCC 2001. Data Compression Conference.

[50]  L. F. Tóth.  Sur la représentation d'une population infinie par un nombre fini d'éléments , 1959 .

[51]  Jean Cardinal.  Design of tree-structured multiple description vector quantizers , 2001, Proceedings DCC 2001. Data Compression Conference.

[52]  P. A. Chou,et al.  When optimal entropy-constrained quantizers have only a finite number of codewords , 1998, Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252).

[53]  Yair Shoham,et al.  Efficient bit allocation for an arbitrary set of quantizers [speech coding] , 1988, IEEE Trans. Acoust. Speech Signal Process..

[54]  Avideh Zakhor,et al.  Bit allocation for joint source/channel coding of scalable video , 2000, IEEE Trans. Image Process..

[55]  Alexandros Eleftheriadis,et al.  Complexity distortion theory , 2003, IEEE Trans. Inf. Theory.

[56]  G. S. Stiles,et al.  Fast full search equivalent encoding algorithms for image compression using vector quantization , 1992, IEEE Trans. Image Process..

[57]  Rudolf Ahlswede,et al.  Network information flow , 2000, IEEE Trans. Inf. Theory.

[58]  R. Ladner.  Entropy-constrained Vector Quantization , 2000 .

[59]  G. R. Walsh,et al.  Methods Of Optimization , 1976 .

[60]  D. Neuhoff,et al.  On The Performance Of Tree-structured Vector Quantization , 1991, Proceedings. 1991 IEEE International Symposium on Information Theory.

[61]  Michael Fleming,et al.  Generalized multiple description vector quantization , 1999, Proceedings DCC'99 Data Compression Conference (Cat. No. PR00096).

[62]  Robert J. Fontana.  Limit theorems for slowly varying composite sources , 1980, IEEE Trans. Inf. Theory.

[63]  Peter N. Yianilos,et al.  Locally lifting the curse of dimensionality for nearest neighbor search (extended abstract) , 2000, SODA '00.

[64]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[65]  Harvey J. Everett.  Generalized Lagrange Multiplier Method for Solving Problems of Optimum Allocation of Resources , 1963 .

[66]  Vinay A. Vaishampayan,et al.  Design of multiple description scalar quantizers , 1993, IEEE Trans. Inf. Theory.

[67]  David L. Neuhoff,et al.  Bennett's integral for vector quantizers , 1995, IEEE Trans. Inf. Theory.

[68]  Jon M. Kleinberg,et al.  Two algorithms for nearest-neighbor search in high dimensions , 1997, STOC '97.

[69]  Michelle Effros.  Practical multi-resolution source coding: TSVQ revisited , 1998, Proceedings DCC '98 Data Compression Conference (Cat. No.98TB100225).

[70]  Lai-Man Po,et al.  Adaptive dimensionality reduction techniques for tree-structured vector quantization , 1994, IEEE Trans. Commun..

[71]  Antonio Ortega,et al.  Complexity-scalable transform coding using variable complexity algorithms , 2000, Proceedings DCC 2000. Data Compression Conference.

[72]  Eve A. Riskin,et al.  Optimal bit allocation via the generalized BFOS algorithm , 1991, IEEE Trans. Inf. Theory.

[73]  Kaizhong Zhang,et al.  A better tree-structured vector quantizer , 1991, [1991] Proceedings. Data Compression Conference.

[74]  Jon Louis Bentley,et al.  An Algorithm for Finding Best Matches in Logarithmic Expected Time , 1977, TOMS.

[75]  Richard E. Blahut,et al.  Computation of channel capacity and rate-distortion functions , 1972, IEEE Trans. Inf. Theory.

[76]  Jianhua Lin.  Multiple-cost constraints for the design of tree-structured vector quantizers , 1995, IEEE Trans. Image Process..

[77]  M. Vetterli,et al.  Computation-distortion characteristics of JPEG encoding and decoding , 1997, Conference Record of the Thirty-First Asilomar Conference on Signals, Systems and Computers (Cat. No.97CB36136).

[78]  Richard E. Ladner,et al.  Fast nearest neighbor search of entropy-constrained vector quantization , 2000, IEEE Trans. Image Process..

[79]  Jean Cardinal,et al.  Tree-Structured Multiple Description Source Coding , 2001 .

[80]  Ulug Bayazit,et al.  Variable-length tree-structured subvector quantization , 1996, Other Conferences.

[81]  Jean Cardinal.  Fast search for entropy-constrained VQ , 1999, Proceedings 10th International Conference on Image Analysis and Processing.

[82]  W. Pearlman,et al.  CONDITIONAL ENTROPY CONSTRAINED TREE STRUCTURED VECTOR QUANTIZATION WITH APPLICATIONS TO SOURCES WITH MEMORY , 1998 .

[83]  Antonio Ortega,et al.  Complexity-distortion tradeoffs in vector matching based on probabilistic partial distance techniques , 1999, Proceedings DCC'99 Data Compression Conference (Cat. No. PR00096).

[84]  Richard E. Ladner,et al.  Nearest neighbor search for data compression , 1999, Data Structures, Near Neighbor Searches, and Methodology.

[85]  Tanya Y. Berger-Wolf,et al.  Optimal Index Assignment for Multichannel Communication (soda 1999, to Appear) , 1998 .

[86]  Allen Gersho,et al.  Optimal nonlinear interpolative vector quantization , 1990, IEEE Trans. Commun..