The Shannon sampling theorem—Its various extensions and applications: A tutorial review

It has been almost thirty years since Shannon introduced the sampling theorem to communications theory. In this review paper we will attempt to present the various contributions made for the sampling theorems with the necessary mathematical details to make it self-contained. We will begin by a clear statement of Shannon's sampling theorem followed by its applied interpretation for time-invariant systems. Then we will review its origin as Whittaker's interpolation series. The extensions will include sampling for functions of more than one variable, random processes, nonuniform sampling, nonband-limited functions, implicit sampling, generalized functions (distributions), sampling with the function and its derivatives as suggested by Shannon in his original paper, and sampling for general integral transforms. Also the conditions on the functions to be sampled will be summarized. The error analysis of the various sampling expansions, including specific error bounds for the truncation, aliasing, jitter and parts of various other errors will be discussed and summarized. This paper will be concluded by searching the different recent applications of the sampling theorems in other fields, besides communications theory. These include optics, crystallography, time-varying systems, boundary value problems, spline approximation, special functions, and the Fourier and other discrete transforms.

[1]  Edmund Taylor Whittaker.  XVIII.—On the Functions which are represented by the Expansions of the Interpolation-Theory , 1915 .

[2]  E. C. Titchmarsh.  The Zeros of Certain Integral Functions , 1926 .

[3]  Harry Nyquist.  Certain Topics in Telegraph Transmission Theory , 1928 .

[4]  J. M. Whittaker.  The “Fourier” Theory of the Cardinal Function , 1928 .

[5]  N. Levinson.  Gap and Density Theorems , 1940 .

[6]  I. J. Schoenberg.  Contributions to the problem of approximation of equidistant data by analytic functions. Part A. On the problem of smoothing or graduation. A first class of analytic approximation formulae , 1946 .

[7]  Lotfi A. Zadeh,et al.  A general theory of linear signal transmission systems , 1952 .

[8]  A. Kohlenberg.  Exact Interpolation of Band‐Limited Functions , 1953 .

[9]  G. D. Francia.  Resolving Power and Information , 1955 .

[10]  Lawrence J. Fogel,et al.  A note on the sampling theorem , 1955, IRE Trans. Inf. Theory.

[11]  Lawrence J. Fogel,et al.  Some general aspects of the sampling theorem , 1956, IRE Trans. Inf. Theory.

[12]  J. Yen.  On Nonuniform Sampling of Bandwidth-Limited Signals , 1956 .

[13]  G. D. Francia.  Directivity, super-gain and information , 1956 .

[14]  R. M. Stewart.  Statistical Design and Evaluation of Filters for the Restoration of Sampled Data , 1956, Proceedings of the IRE.

[15]  R. Bracewell.  Two-dimensional Aerial Smoothing in Radio Astronomy , 1956 .

[16]  A. V. Balakrishnan.  A note on the sampling principle for continuous signals , 1957, IRE Trans. Inf. Theory.

[17]  C. Cahn,et al.  On the sampling the zeros of bandwidth limited signals , 1958, IRE Trans. Inf. Theory.

[18]  H. Kramer,et al.  A Generalized Sampling Theorem , 1959 .

[19]  D. Linden.  A Discussion of Sampling Theorems , 1959, Proceedings of the IRE.

[20]  S. P. Lloyd.  A sampling theorem for stationary (wide sense) stochastic processes. , 1959 .

[21]  John C. Hancock,et al.  A relation between zero-crossings and Fourier coefficients for bandwidth-limited functions (Corresp.) , 1960, IRE Transactions on Information Theory.

[22]  Jr. J. Spilker.  Theoretical Bounds on the Performance of Sampled Data Communications Systems , 1960 .

[23]  Norman M. Abramson,et al.  A Generalization of the Sampling Theorem , 1960, Inf. Control..

[24]  D. Ruchkin,et al.  Linear Reconstruction of Quantized and Sampled Random Signals , 1961 .

[25]  S. Chang.  Optimum transmission of continuous signal over a sampled data link , 1961, Transactions of the American Institute of Electrical Engineers, Part II: Applications and Industry.

[26]  Frederick J. Beutler,et al.  Sampling Theorems and Bases in a Hilbert Space , 1961, Inf. Control..

[27]  W. M. Brown.  Optimum prefiltering of sampled data (Corresp.) , 1961, IRE Trans. Inf. Theory.

[28]  D. Slepian,et al.  Prolate spheroidal wave functions, fourier analysis and uncertainty — II , 1961 .

[29]  David Middleton,et al.  Sampling and Reconstruction of Wave-Number-Limited Functions in N-Dimensional Euclidean Spaces , 1962, Inf. Control..

[30]  Carmen J. Palermo,et al.  System performance in the presence of stochastic delays , 1962, IRE Trans. Inf. Theory.

[31]  A. V. Balakrishnan,et al.  On the problem of time jitter in sampling , 1962, IRE Trans. Inf. Theory.

[32]  J. B. Thomas,et al.  Truncation Error of Sampling-Theorem Expansions , 1962, Proceedings of the IRE.

[33]  D. Middleton,et al.  A Note on Optimum Pre-Sampling Filters , 1963 .

[34]  D. Middleton,et al.  On representative observations , 1963 .

[35]  William M. Brown.  Sampling with Random Jitter , 1963 .

[36]  W. D. Montgomery.  The Gradient in the Sampling of N-dimensional Band-limited Functions† , 1964 .

[37]  David Middleton,et al.  Reconstruction of Multidimensional Stochastic Fields from Discrete Measurements of Amplitude and Gradient , 1964, Inf. Control..

[38]  R. Barakat.  Application of the Sampling Theorem to Optical Diffraction Theory , 1964 .

[39]  Theodore P. Higgins.  A Hypergeometric Function Transform , 1964 .

[40]  L. L. Campbell.  A Comparison of the Sampling Theorems of Kramer and Whittaker , 1964 .

[41]  J. Tukey,et al.  An algorithm for the machine calculation of complex Fourier series , 1965 .

[42]  Moshe Zakai,et al.  Band-Limited Functions and the Sampling Theorem , 1965, Inf. Control..

[43]  Bede Liu,et al.  Sampling representations and the optimum reconstruction of signals , 1965, IEEE Trans. Inf. Theory.

[44]  ON BANDLIMITED PROPERTIES OF FOURIER TRANSFORM PAIRS OF SOME SPECIAL FUNCTIONS , 1965 .

[45]  David Middleton,et al.  Linear interpolation, extrapolation, and prediction of random space-time fields with a limited domain of measurement , 1965, IEEE Trans. Inf. Theory.

[46]  Igor Kluvánek,et al.  Sampling Theorem in Abstract Harmonic Analysis , 1965 .

[47]  A. V. Balakrishnan.  Essentially band-limited stochastic processes (Corresp.) , 1965, IEEE Trans. Inf. Theory.

[48]  W. D. Montgomery.  K-order Sampling of N-dimensional Band-limited Functions† , 1965 .

[49]  O. Leneman,et al.  On "Mean-square reconstruction error" , 1966 .

[50]  O. Leneman,et al.  A note on reconstruction for randomly sampled data , 1966 .

[51]  H. Voelcker.  Toward a unified theory of modulation part I: Phase-envelope relationships , 1966 .

[52]  Frederick J. Beutler,et al.  Random Sampling of Random Processes: Stationary Point Processes , 1966, Inf. Control..

[53]  F. Beutler.  Error-Free Recovery of Signals from Irregularly Spaced Samples , 1966 .

[54]  D. Jagerman,et al.  Bounds for Truncation Error of the Sampling Expansion , 1966 .

[55]  John B. Thomas,et al.  On Truncation Error Bounds for Sampling Representations of Band-Limited Signals , 1966, IEEE Transactions on Aerospace and Electronic Systems.

[56]  Reese T. Prosser.  A multidimensional sampling theorem , 1966 .

[57]  Oscar A.Z. Leneman.  Random sampling of random processes - Optimum linear interpolation. , 1966 .

[59]  A. Papoulis,et al.  Error analysis in sampling theory , 1966 .

[60]  O. Leneman,et al.  Random sampling of random processes: Mean-square comparison of various interpolators , 1966 .

[61]  D. C. Stickler.  An upper bound on aliasing error , 1967 .

[62]  H. Landau.  Sampling, data transmission, and the Nyquist rate , 1967 .

[63]  C. J. Standish.  Two remarks on the reconstruction of sampled non-bandlimited functions , 1967 .

[64]  Kung Yao,et al.  General methods for the derivation of sampling theorems , 1967, IEEE Trans. Inf. Theory.

[65]  Eiiti Takizawa,et al.  On the Generalized Sampling Theorem (非線型振動理論の研究会報告集) , 1968 .

[66]  A. Papoulis.  Limits on bandlimited signals , 1967 .

[67]  Z. A. Piranashvili.  On the Problem of Interpolation of Random Processes , 1967 .

[68]  Kung Yao,et al.  Applications of Reproducing Kernel Hilbert Spaces-Bandlimited Signal Models , 1967, Inf. Control..

[69]  J. L. Brown,et al.  On the error in reconstructing a non-bandlimited function by means of the bandpass sampling theorem , 1967 .

[70]  I. J. Good.  The Loss of Information due to Clipping a Waveform , 1967, Inf. Control..

[71]  A. Papoulis.  Truncated sampling expansions , 1967 .

[72]  J. Cooley,et al.  Application of the fast Fourier transform to computation of Fourier integrals, Fourier series, and convolution integrals , 1967, IEEE Transactions on Audio and Electroacoustics.

[73]  Kazuo Horiuchi,et al.  Sampling Principle for Continuous Signals with Time-Varying Bands , 1968, Inf. Control..

[74]  L. Campbell.  Sampling Theorem for the Fourier Transform of a Distribution with Bounded Support , 1968 .

[75]  Jr. J.L. Brown.  A least upper bound for aliasing error , 1968 .

[76]  Daniel P. Petersen,et al.  On the concept and implementation of sequential analysis for linear random fields , 1968 .

[77]  John L. Brown.  Sampling theorem for finite-energy signals (Corresp.) , 1968, IEEE Trans. Inf. Theory.

[78]  John L. Brown.  Bounds for truncation error in sampling expansions of band-limited signals , 1969, IEEE Trans. Inf. Theory.

[79]  A. J. Jerri.  APPLICATION OF SOME INTERPOLATING FUNCTIONS IN PHYSICS. , 1969 .

[80]  A. J. Jerri,et al.  On the equivalence of Kramer's and Shannon's sampling theorems (Corresp.) , 1969, IEEE Trans. Inf. Theory.

[81]  I. J. Schoenberg,et al.  Cardinal interpolation and spline functions , 1969 .

[82]  P. Lewis,et al.  The finite Fourier transform , 1969 .

[83]  G. Bonnet.  Quelques problèmes d’échantillonnage en traitement linéaire et quadratique des signaux aléatoires , 1969 .

[84]  G. Bonnet.  Corrélateurs hybrides a échantillonnage , 1969 .

[85]  A. J. Jerri.  Some applications for Kramer's generalized sampling theorem , 1969 .

[86]  David Jagerman,et al.  Information theory and approximation of bandlimited functions , 1970, Bell Syst. Tech. J..

[87]  D. P. Petersen.  Discrete and fast Fourier transformations on N-dimensional lattices , 1970 .

[88]  A. Sekey.  A computer simulation study of real-zero interpolation , 1970 .

[89]  Frederick J. Beutler,et al.  Alias-free randomly timed sampling of stochastic processes , 1970, IEEE Trans. Inf. Theory.

[90]  D. Petersen.  Algorithms for Sequential and Random Observations , 1970 .

[91]  S. C. Som.  Simultaneous Multiple Reproduction of Space-Limited Functions by Sampling of Spatial Frequencies*,† , 1970 .

[92]  Ernst Pfaffelhuber,et al.  Sampling series for band-limited generalized functions , 1971, IEEE Trans. Inf. Theory.

[93]  R. Barakat.  Non-linear transformations of stochastic processes associated with Fourier transforms of band-limited positive functions† , 1971 .

[94]  Non-uniform Sampling in Optical Processing , 1971 .

[95]  Holographic restoration of non-uniformly sampled band-limited functions , 1971 .

[96]  L. Ostrander.  The Fourier transform of spline-function approximations to continuous data , 1971 .

[97]  Franco Gori,et al.  Use of non-uniform samplings with a single correcting operation , 1971 .

[98]  William A. Gardner.  A sampling theorem for nonstationary random processes (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[99]  John L. Brown.  Uniform linear prediction of bandlimited processes from past samples (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[100]  John A. Stuller,et al.  Reconstruction of finite duration signals (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[101]  N. Gaarder.  A note on the multidimensional sampling theorem , 1972 .

[102]  P. T. Bason,et al.  An error bound for Lagrange interpolation of low-pass functions (Corresp.) , 1972, IEEE Trans. Inf. Theory.

[103]  A. J. Jerri.  Application of the sampling theorem to time-varying systems , 1972 .

[104]  T. Ericson.  A generalized version of the sampling theorem , 1972 .

[105]  J. R. Higgins.  An Interpolation Series Associated with the Bessel–Hankel Transform , 1972 .

[106]  SUMMATION FORMULAS AND BAND-LIMITED SIGNALS , 1972 .

[107]  R. Kress,et al.  On the General Hermite Cardinal Interpolation , 1972 .

[108]  Joachim Frank,et al.  Computer Processing of Electron Micrographs , 1973 .

[109]  Anthony Ephremides,et al.  On the reconstruction error of sampled data estimates (Corresp.) , 1973, IEEE Trans. Inf. Theory.

[110]  A. J. Jerri.  Sampling theorem for not necessarily finite energy signals , 1973 .

[111]  Henry P Kramer.  The digital form of operators on band-limited functions , 1973 .

[112]  J. J. Hill,et al.  Integral sampling , 1973 .

[113]  L. Rabiner,et al.  A digital signal processing approach to interpolation , 1973 .

[114]  Daniel P. Petersen.  A Comparison of the Performance of Quasi-Optimal and Conventional Objective Analysis Schemes , 1973 .

[115]  Walsh transform of sampled time functions and the sampling principle , 1973 .

[116]  Static and Dynamic Constraints on the Estimation of Space-Time Covariance and Wavenumber-Frequency Spectral Fields , 1973 .

[117]  D. E. Todd.  Sampled data reconstruction of deterministic band-limited signals (Corresp.) , 1973, IEEE Trans. Inf. Theory.

[118]  Amos Nathan.  On Sampling a Function and its Derivatives , 1973, Inf. Control..

[119]  Bhu Dev Sharma,et al.  A Generalized Sampling Theorem for Non-Stationary Random Processes , 1974 .

[120]  Gerald B. Lichtenberger.  A note on perfect predictability and analytic processes (Corresp.) , 1974, IEEE Trans. Inf. Theory.

[121]  F. Joachim.  LETTER TO THE EDITOR: Radiation damage assessment from electron images using digital correlation methods , 1974 .

[122]  A. J. Jerri,et al.  Application of the sampling theorem to boundary value problems , 1974 .

[123]  Sampling theorem and the number of degrees of freedom of an image , 1974 .

[124]  Frederick J. Beutler,et al.  Recovery of Randomly Sampled Signals by Simple Interpolators , 1974, Inf. Control..

[125]  Israel Bar-David,et al.  An Implicit Sampling Theorem for Bounded Bandlimited Functions , 1974, Inf. Control..

[126]  Calculation of upper bounds for errors of an approximate sampled frequency response , 1974 .

[127]  Harvey S. Piper.  Best asymptotic bounds for truncation error in sampling expansions of band-limited signals (Corresp.) , 1975, IEEE Trans. Inf. Theory.

[128]  Sampling Expansions with Derivatives for Finite Hankel and Other Transforms , 1975 .

[129]  Israel Bar-David.  Sample functions of a Gaussian process cannot be recovered from their zero crossings (Corresp.) , 1975, IEEE Trans. Inf. Theory.

[130]  J. Frank,et al.  Signal-to-noise ratio of electron micrographs obtained by cross correlation , 1975, Nature.

[131]  John J. Knab.  System error bounds for Lagrange polynomial estimation of band-limited functions (Corresp.) , 1975, IEEE Trans. Inf. Theory.

[132]  M. Schwartz,et al.  A system error bound for the self-truncating reconstructing filter class (Corresp.) , 1975, IEEE Trans. Inf. Theory.

[133]  Sampling Expansion for Band-Limited Signals Through Some Special Functions , 1975 .

[134]  J. R. Higgins.  A sampling theorem for irregularly spaced sample points (Corresp.) , 1976, IEEE Trans. Inf. Theory.

[135]  Frank Stenger,et al.  Approximations via Whittaker's cardinal function , 1976 .

[136]  M. McDonnell.  A sampling function appropriate for deconvolution (Corresp.) , 1976, IEEE Trans. Inf. Theory.

[137]  John F. Walkup,et al.  A sampling theorem for space-variant systems , 1976 .

[138]  D. Petersen.  Linear sequential coding of random space-time fields , 1976 .

[139]  V. Blazek.  Optical information processing by the Fabry-Perot resonator , 1976 .

[140]  A. J. Jerri.  Sampling expansion for a Languerre-La? transform , 1976 .

[141]  D. Slepian,et al.  On bandwidth , 1976, Proceedings of the IEEE.

[142]  Frederick J. Beutler,et al.  On the truncation error of the cardinal sampling expansion , 1976, IEEE Trans. Inf. Theory.

[143]  Computations of the Hill functions of higher order , 1977 .

[144]  Alan J. Lee.  APPROXIMATE INTERPOLATION AND THE SAMPLING THEOREM , 1977 .

[145]  P. L. Butzer,et al.  A Sampling Theorem for Duration-Limited Functions with Error Estimates , 1977, Inf. Control..