Lip segmentation and tracking under MAP-MRF framework with unknown segment number

This paper proposes a color lip segmentation method with unknown true segment number. Firstly, we build up a multi-layer hierarchical model, in which each layer corresponds to one segment cluster. Subsequently, a Markov random field derived from this model is obtained such that the segmentation problem is formulated as a labeling optimization problem under the maximum a posteriori Markov random field (MAP-MRF) framework. Suppose the pre-assigned number of segment clusters may over-estimate the ground truth, whereby leading to the over-segmentation. We present a rival penalized iterative algorithm capable of performing segment clusters and over-segmentation elimination simultaneously. Based upon this algorithm, we propose a lip segmentation and tracking scheme, featuring the robust performance to the estimate of the number of segment clusters. Experimental results show the efficacy of the proposed method in comparison with the existing counterparts.

[1]  Wen Gao,et al.  Learning and synthesizing MPEG-4 compatible 3-D face animation from video sequence , 2003, IEEE Trans. Circuits Syst. Video Technol..

[2]  Shu Hung Leung,et al.  Automatic lip contour extraction from color images , 2004, Pattern Recognit..

[3]  Jerry L Prince,et al.  Current methods in medical image segmentation. , 2000, Annual review of biomedical engineering.

[4]  Aleix M. Martinez,et al.  The AR face database , 1998 .

[5]  Max K. Agoston,et al.  Computer graphics and geometric modelling - implementation and algorithms , 2005 .

[6]  Léon J. M. Rothkrantz,et al.  Using aerial and geometric features in automatic lip-reading , 2001, INTERSPEECH.

[7]  Jerry L. Prince,et al.  A Survey of Current Methods in Medical Image Segmentation , 1999 .

[8]  Helge Reikeras,et al.  Audio-visual automatic speech recognition using Dynamic Bayesian Networks , 2011 .

[9]  Paul Deléglise,et al.  Statistical Lip-Appearance Models Trained Automatically Using Audio Information , 2002, EURASIP J. Adv. Signal Process..

[10]  N. Otsu.  A threshold selection method from gray level histograms , 1979 .

[11]  Donald Geman,et al.  Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images , 1984, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[12]  Kyuwan Choi,et al.  Detecting the Number of Clusters in n-Way Probabilistic Clustering , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Hadi Seyedarabi,et al.  Automatic Lip Tracking and Action Units Classification using Two-Step Active Contours and Probabilistic Neural Networks , 2006, 2006 Canadian Conference on Electrical and Computer Engineering.

[14]  Christian Wolf,et al.  Inference and parameter estimation on hierarchical belief networks for image segmentation , 2010, Neurocomputing.

[15]  Haluk Derin,et al.  Modeling and Segmentation of Noisy and Textured Images Using Gibbs Random Fields , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[16]  Franc Solina,et al.  COLOR-BASED FACE DETECTION IN THE "15 SECONDS OF FAME" ART INSTALLATION , 2003 .

[17]  Alan Wee-Chung Liew,et al.  Segmentation of color lip images by spatial fuzzy clustering , 2003, IEEE Trans. Fuzzy Syst..

[18]  Kuldip K. Paliwal,et al.  Polynomial features for robust face authentication , 2002, Proceedings. International Conference on Image Processing.

[19]  J. Besag.  Spatial Interaction and the Statistical Analysis of Lattice Systems , 1974 .

[20]  Franck Luthon,et al.  Nonlinear color space and spatiotemporal MRF for hierarchical segmentation of face features in video , 2004, IEEE Transactions on Image Processing.

[21]  Xuelong Li,et al.  A Unified Tensor Level Set for Image Segmentation , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[22]  Alan Wee-Chung Liew,et al.  Fuzzy image clustering incorporating spatial continuity , 2000 .

[23]  Timothy F. Cootes,et al.  Active Shape Models-Their Training and Application , 1995, Comput. Vis. Image Underst..

[24]  Alan Wee-Chung Liew,et al.  Robust lip region segmentation for lip images with complex background , 2007, Pattern Recognit..

[25]  Kotagiri Ramamohanarao,et al.  Automatically Determining the Number of Clusters in Unlabeled Data Sets , 2009, IEEE Transactions on Knowledge and Data Engineering.

[26]  Shu Hung Leung,et al.  Lip image segmentation using fuzzy clustering incorporating an elliptic shape function , 2004, IEEE Transactions on Image Processing.

[27]  Pierre Soille,et al.  Morphological Image Analysis: Principles and Applications , 2003 .

[28]  Yiu-ming Cheung,et al.  Maximum weighted likelihood via rival penalized EM for density mixture clustering with automatic model selection , 2005, IEEE Transactions on Knowledge and Data Engineering.

[29]  Alice Caplier,et al.  Jumping snakes and parametric model for lip segmentation , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[30]  Xuelong Li,et al.  Mammographic mass segmentation: Embedding multiple features in vector-valued level set in ambiguous regions , 2011, Pattern Recognit..

[31]  Jue Wu,et al.  A Segmentation Model Using Compound Markov Random Fields Based on a Boundary Model , 2007, IEEE Transactions on Image Processing.

[32]  Dacheng Tao,et al.  3D human posture segmentation by spectral clustering with surface normal constraint , 2011, Signal Process..

[33]  Timothy F. Cootes,et al.  Extraction of Visual Features for Lipreading , 2002, IEEE Trans. Pattern Anal. Mach. Intell..

[34]  Michael Vogt.  Fast Matching of a Dynamic Lip Model to Color Video Sequences under Regular Illumination Conditions , 1996 .

[35]  Juergen Luettin,et al.  Audio-Visual Automatic Speech Recognition: An Overview , 2004 .

[36]  Timothy F. Cootes,et al.  Active Appearance Models , 2001, IEEE Trans. Pattern Anal. Mach. Intell..

[37]  Zoltan Kato,et al.  Unsupervised segmentation of color textured images using a multilayer MRF model , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[38]  Erkki Oja,et al.  Rival penalized competitive learning for clustering analysis, RBF net, and curve detection , 1993, IEEE Trans. Neural Networks.

[39]  David P. Dobkin,et al.  The quickhull algorithm for convex hulls , 1996, TOMS.

[40]  Juergen Luettin,et al.  Speechreading using Probabilistic Models , 1997, Comput. Vis. Image Underst..

[41]  Donald Geman,et al.  Bayesian Image Analysis , 1986 .

[42]  Tianzi Jiang,et al.  Pixon-based image segmentation with Markov random fields , 2003, IEEE Trans. Image Process..

[43]  A. Murat Tekalp,et al.  Discriminative Analysis of Lip Motion Features for Speaker Identification and Speech-Reading , 2006, IEEE Transactions on Image Processing.

[44]  Vikash Kumar,et al.  A MRF model-based segmentation approach to classification for multispectral imagery , 2002, IEEE Trans. Geosci. Remote. Sens..

[45]  Sridha Sridharan,et al.  An approach to statistical lip modelling for speaker identification via chromatic feature extraction , 1998, Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170).

[46]  Russell M. Mersereau,et al.  Lip feature extraction towards an automatic speechreading system , 2000, Proceedings 2000 International Conference on Image Processing (Cat. No.00CH37101).

[47]  Léon J. M. Rothkrantz,et al.  Mixed Fuzzy-system and Artificial Neural Network Approach to the Automated Recognition of Mouth Expressions , 1998 .

[48]  Juergen Luettin,et al.  Audio-Visual Speech Modeling for Continuous Speech Recognition , 2000, IEEE Trans. Multim..

[49]  Xuelong Li,et al.  A Relay Level Set Method for Automatic Image Segmentation , 2011, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[50]  Walid Mahdi,et al.  Colour and Geometric based Model for Lip Localisation: Application for Lip-reading System , 2007, 14th International Conference on Image Analysis and Processing (ICIAP 2007).

[51]  Yaonan Wang,et al.  A Selection Model for Optimal Fuzzy Clustering Algorithm and Number of Clusters Based on Competitive Comprehensive Fuzzy Evaluation , 2009, IEEE Transactions on Fuzzy Systems.

[52]  Xiangyang Wang,et al.  Color image segmentation using automatic pixel classification with support vector machine , 2011, Neurocomputing.

[53]  Anil K. Jain,et al.  Random field models in image analysis , 1989 .

[54]  Julian Besag,et al.  Digital Image Processing: Towards Bayesian image analysis , 1989 .

[55]  Xuelong Li,et al.  A Review of Active Appearance Models , 2010, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[56]  Zoltan Kato,et al.  A Multi-Layer MRF Model for Video Object Segmentation , 2006, ACCV.

[57]  Julian Besag,et al.  Towards Bayesian image analysis , 1993 .

[58]  Franck Luthon,et al.  Real Time Tracking for 3D Realistic Lip Animation , 2006, 18th International Conference on Pattern Recognition (ICPR'06).

[59]  Stan Z. Li,et al.  Markov Random Field Modeling in Image Analysis , 2001, Computer Science Workbench.