The sampling and reconstruction of time-varying imagery with application in video systems

Sampling is a fundamental operation in all image communication systems. A time-varying image, which is a function of three independent variables, must be sampled in at least two dimensions for transmission over a one-dimensional analog communication channel, and in three dimensions for digital processing and transmission. At the receiver, the sampled image must be interpolated to reconstruct a continuous function of space and time. In imagery destined for human viewing, the visual system forms an integral part of the reconstruction process. This paper presents an overview of the theory of sampling and reconstruction of multidimensional signals. The concept of sampling structures based on lattices is introduced. The important problem of conversion between different sampling structures is also treated. This theory is then applied to the sampling of time-varying imagery, including the role of the camera and display apertures, and the human visual system. Finally, a class of nonlinear interpolation algorithms which adapt to the motion in the scene is presented.

[1]  L. D. Miller.  A New Method of Specifying the Resolving Power of Television Camera Tubes Using the RCA P-300 Test Chart , 1980 .

[2]  R. J. Arguello,et al.  The Effect Of Sampling, Optical Transfer Function Shape, And Anisotropy On Subjective Image Quality , 1982 .

[3]  K. Takikawa.  Simplified 6.3 Mbit/s Codec for Video Conferencing , 1981, IEEE Trans. Commun..

[4]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[5]  T. Chen,et al.  Image Decimation and Interpolation Techniques Based on Frequency Domain Analysis , 1984, IEEE Trans. Commun..

[6]  A H Robinson.  Multidimensional fourier transforms and image processing with finite scanning apertures. , 1973, Applied optics.

[7]  Andreas Polydoros,et al.  Digital interpolation of stochastic signals , 1979 .

[8]  Y. Ninomiya,et al.  A Motion-Compensated Interframe Coding Scheme for Television Pictures , 1982, IEEE Trans. Commun..

[9]  Shaker Sabri,et al.  Movement Compensated Interframe Prediction for NTSC Color TV Signals , 1984, IEEE Trans. Commun..

[10]  B. Prasada,et al.  Low-resolution TV: Subjective effects of frame repetition and picture replenishment , 1967 .

[11]  D. Gabor,et al.  Television band compression by contour interpolation , 1961 .

[12]  B. Baxter,et al.  Rate-Distortion Coding Simulation for Color Images , 1977, IEEE Trans. Commun..

[13]  Ciro Cafforio,et al.  Methods for measuring small displacements of television images , 1976, IEEE Trans. Inf. Theory.

[14]  D. H. Kelly.  Motion and vision. II. Stabilized spatio-temporal threshold surface. , 1979, Journal of the Optical Society of America.

[15]  R. Mersereau,et al.  The processing of periodically sampled multidimensional signals , 1983 .

[16]  J. Ziman.  Principles of the Theory of Solids , 1965 .

[17]  P. Mertz,et al.  A theory of scanning and its relation to the characteristics of the transmitted signal in telephotography and television , 1934 .

[18]  G. J. Tonge.  The Television Scanning Process , 1984 .

[19]  Broder Wendland.  Extended Definition Television with High Picture Quality , 1983 .

[20]  David Middleton,et al.  Sampling and Reconstruction of Wave-Number-Limited Functions in N-Dimensional Euclidean Spaces , 1962, Inf. Control..

[21]  W. Pratt.  Spatial Transform Coding of Color Images , 1971 .

[22]  William B. Thompson,et al.  Lower-Level Estimation and Interpretation of Visual Motion , 1981, Computer.

[23]  J. D. Robbins,et al.  Motion-compensated television coding: Part I , 1979, The Bell System Technical Journal.

[24]  Francis Kretz,et al.  Échantillonnage des images de télévision: analyse dans le domaine spatio-temporel et dans le domaine de Fourier , 1981 .

[25]  Anil K. Jain,et al.  Displacement Measurement and Its Application in Interframe Image Coding , 1981, IEEE Trans. Commun..

[26]  J.-Y. Ouellet,et al.  Three-Dimensional Spectrum and Processing of Digital NTSC Color Signals , 1980 .

[27]  Earl F. Brown,et al.  Low-resolution TV: Subjective comparison of interlaced and noninterlaced pictures , 1967 .

[28]  J.-Y. Ouellet,et al.  Sampling and Reconstruction of NTSC Video Signals at Twice the Color Subcarrier Frequency , 1981, IEEE Trans. Commun..

[29]  Toshio Koga,et al.  Motion-Adaptive Interpolation for Videoconference Pictures , 1984, ICC.

[30]  R.N. Bracewell,et al.  Signal analysis , 1978, Proceedings of the IEEE.

[31]  Eric Dubois,et al.  Noise Reduction in Image Sequences Using Motion-Compensated Temporal Filtering , 1984, IEEE Trans. Commun..

[32]  Hsieh Hou,et al.  Cubic splines for image interpolation and digital filtering , 1978 .

[33]  S. Biyiksiz,et al.  Multirate digital signal processing , 1985, Proceedings of the IEEE.

[34]  S. Sabri,et al.  Coding of Video Signals at 50 Kb/s using Motion Compensation Techniques , 1983, MILCOM 1983 - IEEE Military Communications Conference.

[35]  R. Paquin,et al.  A spatio-temporal gradient method for estimating the displacement field in time-varying imagery , 1982, Computer Graphics and Image Processing.

[36]  D. E. Pearson,et al.  Transmission and display of pictorial information , 1975 .

[37]  Otto H. Schade.  Image Reproduction by a Line Raster Process , 1973 .

[38]  Takahiko Fukinuki,et al.  A Motion-Adaptive High-Definition Converter for NTSC Color TV Signals , 1984 .

[39]  John Newland Ratzel.  The discrete representation of spatially continuous images , 1980 .

[40]  Jan P. Allebach.  Design of antialiasing patterns for time-sequential sampling of spatiotemporal signals , 1984 .

[41]  W. Huggins,et al.  Signal Theory , 1956 .

[42]  F. Davidoff.  Digital television coding standards , 1982 .

[43]  A Macovski.  Spatial and temporal analysis of scanned systems. , 1970, Applied optics.

[44]  O. Faugeras.  Digital color image processing within the framework of a human visual model , 1979 .

[45]  C. A. Rogers,et al.  An Introduction to the Geometry of Numbers , 1959 .

[46]  C. P. Sandbank,et al.  High-Definition Television and Compatibility with Existing Standards , 1982 .

[47]  Thomas W. Parks,et al.  New results in the design of digital interpolators" ieee trans , 1975 .

[48]  Z. L. Budrikis,et al.  Picture Quality Prediction Based on a Visual Model , 1982, IEEE Trans. Commun..

[49]  W. Rudin,et al.  Fourier Analysis on Groups. , 1965 .

[50]  G. A. Reitmeier.  The Effects of Analog Filtering on the Picture Quality of Component Digital Television Systems , 1981 .

[51]  N. Gaarder.  A note on the multidimensional sampling theorem , 1972 .