Multimodal Information Fusion

Publisher Summary This chapter gives an overview of multimodal information fusion from the machine-learning perspective. Humans interact with each other using different modalities of communication. These include speech, gestures, documents, etc. It is therefore natural that human–computer interaction (HCI) should facilitate the same multimodal form of communication. To capture this information, one uses different types of sensors, i.e., microphones to capture the audio signal, cameras to capture life video images, 3D sensors to directly capture the surface information in real time. In each of these cases, commercial off-the-shelf (COTS) devices are already available and can be readily deployed for HCI applications. Examples of HCI applications include audio-visual speech recognition, gesture recognition, emotional recognition, and person recognition using biometrics.

[1]  Josef Kittler,et al.  On Combination of Face Authentication Experts by a Mixture of Quality Dependent Fusion Classifiers , 2007, MCS.

[2]  Javier Ortega-Garcia,et al.  Kernel-based multimodal biometric verification using quality signals , 2004, SPIE Defense + Commercial Sensing.

[3]  Roland Auckenthaler,et al.  Score Normalization for Text-Independent Speaker Verification Systems , 2000, Digit. Signal Process..

[4]  Josef Kittler,et al.  Quality dependent fusion of intramodal and multimodal biometric experts , 2007, SPIE Defense + Commercial Sensing.

[5]  Alexander J. Smola,et al.  Advances in Large Margin Classifiers , 2000 .

[6]  Gérard Chollet,et al.  Comparing decision fusion paradigms using -NN based classifiers, decision trees and logistic regression in a multi-modal identity verification ap plication , 1999 .

[7]  Tsuhan Chen,et al.  Audio-visual integration in multimodal communication , 1998, Proc. IEEE.

[8]  Ching Y. Suen,et al.  A Method of Combining Multiple Experts for the Recognition of Unconstrained Handwritten Numerals , 1995, IEEE Trans. Pattern Anal. Mach. Intell..

[9]  Krzysztof Kryszczuk,et al.  Improving Classification with Class-Independent Quality Measures: Q-stack in Face Verification , 2007, ICB.

[10]  Arun Ross,et al.  Score normalization in multimodal biometric systems , 2005, Pattern Recognit..

[11]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[12]  Francis Jack Smith,et al.  Speech recognition with unknown partial feature corruption - a review of the union model , 2003, Comput. Speech Lang..

[13]  Julian Fierrez,et al.  Dealing with sensor interoperability in multi-biometrics: the UPM experience at the Biosecure Multimodal Evaluation 2007 , 2008, SPIE Defense + Commercial Sensing.

[14]  Samy Bengio,et al.  Improving Fusion with Margin-Derived Confidence in Biometric Authentication Tasks , 2005, AVBPA.

[15]  Terrance E. Boult,et al.  Classification Enhancement via Biometric Pattern Perturbation , 2005, AVBPA.

[16]  Julian Fiérrez,et al.  Multimodal biometric authentication using quality signals in mobile communications , 2003, 12th International Conference on Image Analysis and Processing, 2003.Proceedings..

[17]  Arun Ross,et al.  Image versus feature mosaicing: a case study in fingerprints , 2006, SPIE Defense + Commercial Sensing.

[18]  T. Bourlai,et al.  Improving Biometric Device Interoperability by Likelihood Ratio-based Quality Dependent Score Normalization , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[19]  Kuldip K. Paliwal,et al.  Information Fusion and Person Verification Using Speech & Face Information , 2002 .

[20]  John P. Baker,et al.  Fusing multimodal biometrics with quality estimates via a Bayesian belief network , 2008, Pattern Recognit..

[21]  Ying Zhu,et al.  Multi-Camera Face Recognition by Reliability-Based Selection , 2006, 2006 IEEE International Conference on Computational Intelligence for Homeland Security and Personal Safety.

[22]  Norman Poh,et al.  Multi-system Biometric Authentication: Optimal Fusion and User-Specific Information , 2006 .

[23]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[24]  Conrad Sanderson,et al.  Automatic Person Verification Using Speech and Face Information , 2003 .

[25]  Anil K. Jain,et al.  A Principled Approach to Score Level Fusion in Multimodal Biometric Systems , 2005, AVBPA.

[26]  Wei-Yun Yau,et al.  Fusion of Auxiliary Information for Multi-modal Biometrics Authentication , 2004, ICBA.

[27]  Craig I. Watson,et al.  Studies of biometric fusion , 2006 .

[28]  A. Ross,et al.  Level Fusion Using Hand and Face Biometrics , 2005 .

[29]  Simon Lucey,et al.  Audio-visual Speech Processing , 2002 .

[30]  Arun Ross,et al.  Handbook of Multibiometrics , 2006, The Kluwer international series on biometrics.

[31]  Karthik Nandakumar,et al.  INTEGRATION OF MULTIPLE CUES IN BIOMETRIC SYSTEMS , 2005 .

[32]  Arun Ross,et al.  Feature level fusion of hand and face biometrics , 2005, SPIE Defense + Commercial Sensing.

[33]  Subhash C. Bagui,et al.  Combining Pattern Classifiers: Methods and Algorithms , 2005, Technometrics.

[34]  Krzysztof Kryszczuk,et al.  Reliability-Based Decision Fusion in Multimodal Biometric Verification Systems , 2007, EURASIP J. Adv. Signal Process..

[35]  Josef Kittler,et al.  Quality Controlled Multimodal Fusion of Biometric Experts , 2007, CIARP.

[36]  Jonas Richiardi,et al.  Probabilistic models for multi-classifier biometric authentication using quality measures , 2008 .

[37]  Sargur N. Srihari,et al.  Decision Combination in Multiple Classifier Systems , 1994, IEEE Trans. Pattern Anal. Mach. Intell..

[38]  Juergen Luettin,et al.  Visual Speech and Speaker Recognition , 1997 .

[39]  M. L. Shire,et al.  Discriminant Training of Front-End and Acoustic Modeling Stages to Heterogeneous Acoustic Environmen , 2000 .

[40]  Xiang Gao,et al.  Predicting biometric system failure , 2005, CIHSPS 2005. Proceedings of the 2005 IEEE International Conference on Computational Intelligence for Homeland Security and Personal Safety, 2005..

[41]  Anil K. Jain,et al.  Likelihood Ratio-Based Biometric Score Fusion , 2008, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[42]  Arun Ross,et al.  Performance enhancement of 2D face recognition via mosaicing , 2005, Fourth IEEE Workshop on Automatic Identification Advanced Technologies (AutoID'05).

[43]  Adam Krzyżak,et al.  Methods of combining multiple classifiers and their applications to handwriting recognition , 1992, IEEE Trans. Syst. Man Cybern..

[44]  Yillbyung Lee,et al.  Biometric Authentication System Using Reduced Joint Feature Vector of Iris and Face , 2005, AVBPA.

[45]  Ching Y. Suen,et al.  Application of majority voting to pattern recognition: an analysis of its behavior and performance , 1997, IEEE Trans. Syst. Man Cybern. Part A.

[46]  Gavin Brown,et al.  Diversity in neural network ensembles , 2004 .

[47]  Astrid Hagen.  Robust speech recognition based on multi-stream processing , 2001 .

[48]  Krzysztof Kryszczuk,et al.  Classification with class-independent quality information for biometric verification , 2008 .