Next-generation content representation, creation, and searching for new-media applications in education

Content creation, editing, and searching are extremely time-consuming tasks that often require substantial training and experience, especially when high-quality audio and video are involved. New media represents a new paradigm for multimedia information representation and processing, in which the emphasis is placed on the actual content. It thus brings the tasks of content creation and searching much closer to actual users and enables them to be active producers of audio-visual information rather than passive recipients. We discuss the state of the art and present next-generation techniques for content representation, searching, creation and editing. We discuss our experiences in developing a Web-based distributed compressed video editing and searching system (WebClip), a media-representation language (Flavor) and an object-based video authoring system (Zest) based on it, and a large image/video search engine for the World Wide Web (WebSEEk). We also present a case study of new media applications based on specific planned multimedia education experiments with the above systems in several K-12 schools in Manhattan, NY.

[1]  Alexandros Eleftheriadis.  The MPEG-4 system and description languages: from practice to theory , 1997, Proceedings of 1997 IEEE International Symposium on Circuits and Systems. Circuits and Systems in the Information Age ISCAS '97.

[2]  H. Hoffmann.  EBU/SMPTE joint task force for harmonised standards for the exchange of television programmes as bit streams-a progress report , 1997 .

[3]  Thomas A. Ohanian,et al.  Digital Nonlinear Editing: New Approaches to Editing Film and Video , 1993 .

[4]  Clu-istos Foutsos,et al.  Fast subsequence matching in time-series databases , 1994, SIGMOD '94.

[5]  Jelena Kovacevic,et al.  Wavelets and Subband Coding , 2013, Prentice Hall Signal Processing Series.

[6]  Steven K. Feiner,et al.  Introduction to Computer Graphics , 1993 .

[7]  David R. Nadeau,et al.  The Vrml Sourcebook , 1996 .

[8]  Dragutin Petkovic,et al.  Query by Image and Video Content: The QBIC System , 1995, Computer.

[9]  Shih-Fu Chang,et al.  Clustering methods for video browsing and annotation , 1996, Electronic Imaging.

[10]  Shih-Fu Chang,et al.  Development of Columbia's video on demand testbed , 1996, Signal Process. Image Commun..

[11]  Lyman P. Hurd,et al.  Fractal image compression , 1993 .

[12]  Arun N. Netravali,et al.  Digital Video: An introduction to MPEG-2 , 1996 .

[13]  Takeo Kanade,et al.  Development of a video-rate stereo machine , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[14]  Paul M. B. Vitányi,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1993, Graduate Texts in Computer Science.

[15]  Rosalind W. Picard,et al.  Interactive Learning Using a "Society of Models" , 2017, CVPR 1996.

[16]  Shree K. Nayar,et al.  Catadioptric omnidirectional camera , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[17]  Allen Gersho,et al.  Vector quantization and signal compression , 1991, The Kluwer international series in engineering and computer science.

[18]  Myron Flickner,et al.  Query by Image and Video Content , 1995 .

[19]  Ray A. Jarvis,et al.  A Perspective on Range Finding Techniques for Computer Vision , 1983, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[20]  Rakesh Mohan,et al.  Text-based search of TV news stories , 1996, Other Conferences.

[21]  Boon-Lock Yeo,et al.  Video content characterization and compaction for digital library applications , 1997, Electronic Imaging.

[22]  T.S. Huang,et al.  A relevance feedback architecture for content-based multimedia information retrieval systems , 1997, 1997 Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries.

[23]  Boon-Lock Yeo,et al.  Rapid scene analysis on compressed video , 1995, IEEE Trans. Circuits Syst. Video Technol..

[24]  David Salesin,et al.  Fast multiresolution image querying , 1995, SIGGRAPH.

[25]  Antonin Guttman,et al.  R-trees: a dynamic index structure for spatial searching , 1984, SIGMOD '84.

[26]  David A. Forsyth,et al.  Body plans , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[27]  Amarnath Gupta,et al.  Virage image search engine: an open framework for image management , 1996, Electronic Imaging.

[28]  Arif Ghafoor,et al.  Spatio-temporal composition of distributed multimedia objects for value-added networks , 1991, Computer.

[29]  Joan L. Mitchell,et al.  JPEG: Still Image Data Compression Standard , 1992 .

[30]  John R. Smith,et al.  Searching for Images and Videos on the World-Wide Web , 1999 .

[31]  Shih-Fu Chang,et al.  Visually Searching the Web for Content , 1997, IEEE Multim..

[32]  Stephen W. Smoliar,et al.  Content based video indexing and retrieval , 1994, IEEE MultiMedia.

[33]  Adele E. Howe,et al.  Experiences with selecting search engines using metasearch , 1997, TOIS.

[34]  Shih-Fu Chang,et al.  Video on Demand Systems: Technology, Interoperability, and Trials , 1997 .

[35]  Jerry D. Gibson,et al.  Digital coding of waveforms: Principles and applications to speech and video , 1985, Proceedings of the IEEE.

[36]  Shih-Fu Chang,et al.  Enhancing image search engines in visual information environments , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[37]  Michael Stonebraker,et al.  Chabot: Retrieval from a Relational Database of Images , 1995, Computer.

[38]  Guy L. Steele,et al.  The Java Language Specification , 1996 .

[39]  Andrew S. Glassner,et al.  Principles of Digital Image Synthesis , 1995 .

[40]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[41]  Shih-Fu Chang,et al.  Visual information retrieval from large distributed online repositories , 1997, CACM.

[42]  Fernando Pereira,et al.  MPEG-4: Context and objectives , 1997, Signal Process. Image Commun..

[43]  Shih-Fu Chang,et al.  Manipulation and Compositing of MC-DCT Compressed Video , 1995, IEEE J. Sel. Areas Commun..

[44]  Minoru Etoh,et al.  MPEG-4, part 1: Invited papers , 1997, Signal Process. Image Commun..

[45]  Alexander G. Hauptmann,et al.  Text, Speech, and Vision for Video Segmentation: The InformediaTM Project , 1995 .

[46]  Euripides G. M. Petrakis,et al.  Similarity Searching in Medical Image Databases , 1997, IEEE Trans. Knowl. Data Eng..

[47]  Kathy Kozel.  The Classes of Authoring Programs. , 1997 .

[48]  Jerome M. Shapiro,et al.  Embedded image coding using zerotrees of wavelet coefficients , 1993, IEEE Trans. Signal Process..

[49]  B. Wilson.  Constructivist Learning Environments: Case Studies in Instructional Design , 1995 .

[50]  松田 晃一,et al.  Virtual Reality Modeling Language , 1997 .

[51]  Behzad Shahraray,et al.  Automatic generation of pictorial transcripts of video programs , 1995, Electronic Imaging.

[52]  Sara Shatford,et al.  Analyzing the Subject of a Picture: A Theoretical Approach , 1986 .

[53]  Kim A. Kastens,et al.  An Interactive Multimedia Tool for Helping Students “Translate” from Maps to Reality and Vice Versa , 1996 .

[54]  Jing Huang,et al.  Combining supervised learning with color correlograms for content-based image retrieval , 1997, MULTIMEDIA '97.

[55]  Shih-Fu Chang,et al.  Compressed-domain techniques for image/video indexing and manipulation , 1995, Proceedings., International Conference on Image Processing.

[56]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[57]  Shih-Fu Chang,et al.  A distributed system for editing and browsing compressed video over the network , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[58]  James Lee Hafner,et al.  Efficient Color Histogram Indexing for Quadratic Form Distance Functions , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[59]  Brian Christopher Smith,et al.  A resolution independent video language , 1995, MULTIMEDIA '95.

[60]  Hans-Peter Kriegel,et al.  The R*-tree: an efficient and robust access method for points and rectangles , 1990, SIGMOD '90.

[61]  Alexandros Eleftheriadis,et al.  Flavor: a language for media representation , 1997, MULTIMEDIA '97.

[62]  Philip A. Chou,et al.  The MPEG-4 systems and description languages: A way ahead in audio visual information representation , 1997, Signal Process. Image Commun..

[63]  Thomas P. Minka,et al.  An image database browser that learns from user interaction , 1996 .

[64]  Karen Spärck Jones,et al.  Open-vocabulary speech indexing for voice and video mail retrieval , 1997, MULTIMEDIA '96.

[65]  Minoru Etoh,et al.  MPEG-4, Part 2: Submitted papers , 1997, Signal Process. Image Commun..

[66]  P. Anandan,et al.  Interactive content-based video indexing and browsing , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[67]  Shih-Fu Chang,et al.  MetaSEEk: a content-based metasearch engine for images , 1997, Electronic Imaging.

[68]  Peter No,et al.  Digital Coding of Waveforms , 1986 .

[69]  Alex Pentland,et al.  Photobook: tools for content-based manipulation of image databases , 1994, Electronic Imaging.

[70]  Tom Minka,et al.  Interactive learning with a "society of models" , 1997, Pattern Recognit..

[71]  Shree K. Nayar,et al.  Real-time focus range sensor , 1995, Proceedings of IEEE International Conference on Computer Vision.

[72]  Shih-Fu Chang,et al.  VideoQ: an automated content based video search system using visual cues , 1997, MULTIMEDIA '97.

[73]  Jie Liang.  Highly scalable image coding for multimedia applications , 1997, MULTIMEDIA '97.

[74]  John B. Black,et al.  An Interpretation Construction Approach to Constructivist Design , 2000 .

[75]  Shih-Fu Chang,et al.  WebClip: a WWW video editing/browsing system , 1997, Proceedings of First Signal Processing Society Workshop on Multimedia Signal Processing.

[76]  Rohini K. Srihari,et al.  Automatic Indexing and Content-Based Retrieval of Captioned Images , 1995, Computer.

[77]  Shih-Fu Chang,et al.  VisualSEEk: a fully automated content-based image query system , 1997, MULTIMEDIA '96.

[78]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[79]  Alexandros Eleftheriadis,et al.  A syntactic framework for bitstream-level representation of audio-visual objects , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[80]  Arun N. Netravali,et al.  Digital Pictures: Representation, Compression and Standards , 1995 .

[81]  Shih-Fu Chang,et al.  CVEPS - a compressed video editing and parsing system , 1997, MULTIMEDIA '96.

[82]  Stuart Weibel,et al.  Image Description on the Internet: A Summary of the CNI/OCLC Image Metadata Workshop September 24 - 25, 1996, Dublin, Ohio , 1997, D Lib Mag..

[83]  Alberto Del Bimbo,et al.  Visual information retrieval , 1999 .

[84]  Jon Louis Bentley,et al.  An Algorithm for Finding Best Matches in Logarithmic Expected Time , 1977, TOMS.

[85]  Michael J. Swain,et al.  WebSeer: An Image Search Engine for the World Wide Web , 1996 .