A Bayesian network-based framework for semantic image understanding

Current research in content-based semantic image understanding is largely confined to exemplar-based approaches built on low-level feature extraction and classification. The ability to extract both low-level and semantic features and perform knowledge integration of different types of features is expected to raise semantic image understanding to a new level. Belief networks, or Bayesian networks (BN), have proven to be an effective knowledge representation and inference engine in artificial intelligence and expert systems research. Their effectiveness is due to the ability to explicitly integrate domain knowledge in the network structure and to reduce a joint probability distribution to conditional independence relationships. In this paper, we present a general-purpose knowledge integration framework that employs BN in integrating both low-level and semantic features. The efficacy of this framework is demonstrated via three applications involving semantic understanding of pictorial images. The first application aims at detecting main photographic subjects in an image, the second aims at selecting the most appealing image in an event, and the third aims at classifying images into indoor or outdoor scenes. With these diverse examples, we demonstrate that effective inference engines can be built within this powerful and flexible framework according to specific domain knowledge and available training data to solve inherently uncertain vision problems.

[1]  Anil K. Jain,et al.  Content-based hierarchical classification of vacation images , 1999, Proceedings IEEE International Conference on Multimedia Computing and Systems.

[2]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems - networks of plausible inference , 1991, Morgan Kaufmann series in representation and reasoning.

[3]  Jiebo Luo,et al.  Efficient Mobile Imaging Using Emphasis Image Selection , 2003, PICS.

[4]  W. Eric L. Grimson,et al.  Configuration based scene classification and image indexing , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[5]  Jiebo Luo,et al.  Ground truth for training and evaluation of automatic main subject detection , 2000, Electronic Imaging.

[6]  M. Kendall.  Probability and Statistical Inference , 1956, Nature.

[7]  Steffen L. Lauritzen,et al.  Graphical models in R , 1996 .

[8]  Jiebo Luo,et al.  Quantitative evaluation of rank-order similarity of images , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[9]  Thomas S. Huang,et al.  Image processing , 1971 .

[10]  Martin Szummer,et al.  Indoor-outdoor image classification , 1998, Proceedings 1998 IEEE International Workshop on Content-Based Access of Image and Video Database.

[11]  Dana H. Ballard,et al.  Computer Vision , 1982 .

[12]  M. E. Ulug.  The use of fuzzy neural networks for feature/sensor selection , 1994, Proceedings of 1994 IEEE International Conference on MFI '94. Multisensor Fusion and Integration for Intelligent Systems.

[13]  Pamela R. Lipson,et al.  Context and configuration based scene classification , 1996 .

[14]  Moshe Kam,et al.  Sensor Fusion for Mobile Robot Navigation , 1997, Proc. IEEE.

[15]  Glynn P. Robinson,et al.  Model-based recognition of anatomical objects from medical images , 1994, Image Vis. Comput..

[16]  Andreas Savakis,et al.  Bayesian network structure learning and inference in indoor vs. outdoor image classification , 2004, ICPR 2004.

[17]  John R. Smith,et al.  Image Classification and Querying Using Composite Region Templates , 1999, Comput. Vis. Image Underst..

[18]  Anil K. Jain,et al.  Automatic image orientation detection , 1999, Proceedings 1999 International Conference on Image Processing (Cat. 99CH36348).

[19]  Allen Klinger,et al.  Preference voting for sensor fusion , 1990, Defense, Security, and Sensing.

[20]  Anthony J. Maeder,et al.  Automatic identification of perceptually important regions in an image , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[21]  Milan Sonka,et al.  Image Processing, Analysis and Machine Vision , 1993, Springer US.

[22]  Duncan Fyfe Gillies,et al.  Using Hidden Nodes in Bayesian Networks , 1996, Artif. Intell..

[23]  T. Kanade,et al.  Color information for region segmentation , 1980 .

[24]  Lawrence A. Klein,et al.  Sensor and Data Fusion Concepts and Applications , 1993 .

[25]  Wray L. Buntine.  Operations for Learning with Graphical Models , 1994, J. Artif. Intell. Res..

[26]  Judea Pearl,et al.  Probabilistic reasoning in intelligent systems , 1988 .

[27]  Shih-Fu Chang,et al.  A knowledge engineering approach for image classification based on probabilistic reasoning systems , 2000, 2000 IEEE International Conference on Multimedia and Expo. ICME2000. Proceedings. Latest Advances in the Fast Changing World of Multimedia (Cat. No.00TH8532).

[28]  Andreas E. Savakis,et al.  Evaluation of image appeal in consumer photography , 2000, Electronic Imaging.

[29]  Jiebo Luo,et al.  A computational approach to determination of main subject regions in photographic images , 2004, Image Vis. Comput..

[30]  Bernadette Dorizzi,et al.  Neural networks and fuzzy data fusion. Application to an on-line and real time vehicle detection system , 1999, Pattern Recognit. Lett..

[31]  Yongmei Wang,et al.  Content-based image orientation detection with support vector machines , 2001, Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries (CBAIVL 2001).

[32]  Jiebo Luo,et al.  Improved scene classification using efficient low-level features and semantic cues , 2004, Pattern Recognit..

[33]  Gregory M. Provan,et al.  The Sensitivity of Belief Networks to Imprecise Probabilities: An Experimental Investigation , 1996, Artif. Intell..

[34]  Eberhard Mandler,et al.  Document analysis-from pixels to contents , 1992 .

[35]  Sudeep Sarkar,et al.  Modeling Parameter Space Behavior of Vision Systems Using Bayesian Networks , 2000, Comput. Vis. Image Underst..

[36]  Jiebo Luo,et al.  A computationally efficient approach to indoor/outdoor scene classification , 2002, Object recognition supported by user interaction for service robots.

[37]  P. J. Green,et al.  Probability and Statistical Inference , 1978 .

[38]  V Dallos,et al.  Can computer aided teaching packages improve clinical care in patients with acute abdominal pain? , 1991, BMJ.

[39]  Andreas E. Savakis,et al.  Automated event clustering and quality screening of consumer pictures for digital albuming , 2003, IEEE Trans. Multim..

[40]  Anil K. Jain,et al.  On image classification: city images vs. landscapes , 1998, Pattern Recognit..

[41]  Henrik I. Christensen,et al.  Application of voting to fusion of purposive modules: An experimental investigation , 1998, Robotics Auton. Syst..

[42]  Marcel Worring,et al.  Content-Based Image Retrieval at the End of the Early Years , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[43]  Shih-Fu Chang,et al.  Integration of Visual and Text-Based Approaches for the Content Labeling and Classification of Photographs , 1999, SIGIR 1999.

[44]  Robert M. Fung,et al.  Applying Bayesian networks to information retrieval , 1995, CACM.

[45]  David Heckerman,et al.  A Tutorial on Learning with Bayesian Networks , 1998, Learning in Graphical Models.