Instantly indexed multimedia databases of real world events

We introduce a new paradigm for real-time conversion of a real world event into a rich multimedia database by processing data from multiple sensors observing the event. A real-time analysis of the sensor data, tightly coupled with domain knowledge, results in instant indexing of multimedia data at capture time. This yields semantic information to answer complex queries about the content and the ability to extract portions of data that correspond to complex actions performed in the real world. The power of such an instantly indexed multimedia database system, in content-based retrieval of multimedia data or in semantic analysis and visualization of the data, far exceeds that of systems which index multimedia data only after it is produced. We present LucentVision, an instantly indexed multimedia database system developed for the sport of tennis. This system analyzes video from multiple cameras in real time and captures the activity of the players and the ball in the form of motion trajectories. The system stores these trajectories in a database along with video, 3D models of the environment, scores, and other domain-specific information. LucentVision has been used to enhance live television and Internet broadcasts with game analyses and virtual replays in more than 250 international tennis matches.

[1]  K. Rohr.  Towards model-based recognition of human movements in image sequences , 1994 .

[2]  Alex Pentland,et al.  Pfinder: Real-Time Tracking of the Human Body , 1997, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Yoshiaki Shirai,et al.  Detection of the movements of persons from a sparse sequence of TV images , 1983, Pattern Recognition.

[4]  Takeo Kanade,et al.  Virtualized reality: constructing time-varying virtual worlds from real world events , 1997, Proceedings. Visualization '97 (Cat. No. 97CB36155).

[5]  Takeo Kanade,et al.  Virtualized reality: concepts and early results , 1995, Proceedings IEEE Workshop on Representation of Visual Scenes (In Conjunction with ICCV'95).

[6]  Anil K. Jain,et al.  Automatic classification of tennis video for high-level content-based retrieval , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[7]  Jakub Segen,et al.  Performance evaluation of people tracking systems , 1996, Proceedings Third IEEE Workshop on Applications of Computer Vision. WACV'96.

[8]  Yoshinori Kuno,et al.  Multiple object tracking system with three level continuous processes , 1992, [1992] Proceedings IEEE Workshop on Applications of Computer Vision.

[9]  Takeo Kanade,et al.  Virtual ized reality: constructing time-varying virtual worlds from real world events , 1997 .

[10]  R. Y. Tsai,et al.  An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision , 1986, CVPR 1986.

[11]  Jake K. Aggarwal,et al.  Tracking human motion using multiple cameras , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[12]  HongJiang Zhang,et al.  Automatic parsing of TV soccer programs , 1995, Proceedings of the International Conference on Multimedia Computing and Systems.

[13]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[14]  Yves Jean,et al.  Ball tracking and virtual replays for innovative tennis broadcasts , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[15]  Ramesh C. Jain,et al.  An architecture for multiple perspective interactive video , 1995, MULTIMEDIA '95.

[16]  Jun-ichi Hasegawa,et al.  Development of motion analysis system for quantitative evaluation of teamwork in soccer games , 1996, Proceedings of 3rd IEEE International Conference on Image Processing.

[17]  Dieter Pfoser,et al.  Novel Approaches to the Indexing of Moving Object Trajectories , 2000, VLDB.

[18]  Yves Jean,et al.  Visualization of sports using motion trajectories: providing insights into performance, style, and strategy , 2001, Proceedings Visualization, 2001. VIS '01..

[19]  Yves Jean,et al.  Real time tracking for enhanced tennis broadcasts , 1998, Proceedings. 1998 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No.98CB36231).

[20]  David C. Hogg.  Model-based vision: a program to see a walking person , 1983, Image Vis. Comput..

[21]  Markus Schneider,et al.  Developments in spatio-temporal query languages , 1999, Proceedings. Tenth International Workshop on Database and Expert Systems Applications. DEXA 99.

[22]  Jakub Segen,et al.  A camera-based system for tracking people in real time , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[23]  Koichiro Akita,et al.  Image sequence analysis of real world human motion , 1984, Pattern Recognit..

[24]  Jake K. Aggarwal,et al.  Human Motion Analysis: A Review , 1999, Comput. Vis. Image Underst..

[25]  Edward R. Dougherty,et al.  Morphological methods in image and signal processing , 1988 .

[26]  Aaron F. Bobick,et al.  Visual Tracking Using Closed-Worlds , 1995 .

[27]  L. Davis,et al.  el-based tracking of humans in action: , 1996 .

[28]  Mitsuho Yamada,et al.  Development of a high-performance camera-control device and its application to TV programs , 1999 .

[29]  Stephen W. Smoliar,et al.  An integrated system for content-based video retrieval and browsing , 1997, Pattern Recognit..

[30]  Hanspeter Bieri,et al.  SoccerMan-reconstructing soccer games from video sequences , 1998, Proceedings 1998 International Conference on Image Processing. ICIP98 (Cat. No.98CB36269).

[31]  R. F. Rashid,et al.  Towards a system for the interpretation of moving light displays , 1980, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[32]  Timos K. Sellis,et al.  Spatio-temporal indexing for large multimedia applications , 1996, Proceedings of the Third IEEE International Conference on Multimedia Computing and Systems.

[33]  Boon-Lock Yeo,et al.  Video content characterization and compaction for digital library applications , 1997, Electronic Imaging.

[34]  Aaron F. Bobick,et al.  Closed-world tracking , 1995, Proceedings of IEEE International Conference on Computer Vision.

[35]  Gopal Sarma Pingali,et al.  Multimedia retrieval through spatio-temporal activity maps , 2001, MULTIMEDIA '01.

[36]  Paul Douglas.  The Handbook of Tennis , 1982 .