Conformal Predictions for Information Fusion A Comparative Study of P-Value Combination Methods

The increased availability of a wide range of sensing technologies over the last few decades has resulted in an equivalent increased need for reliable information fusion methods in machine learning applications. While existing theories such as the Dempster-Shafer theory and the possibility theory have been used for several years now, they do not provide guarantees of error calibration in information fusion settings. The Conformal Predictions (CP) framework is a new game-theoretic approach to reliable machine learning, which provides a methodology to obtain error calibration under classification and regression settings. In this work, we present a methodology to extend the Conformal Predictions framework to both classification and regression-based information fusion settings. This methodology is based on applying the CP framework to each data source as an independent hypothesis test, and subsequently using p-value combination methods as a test statistic for the combined hypothesis after fusion. The proposed methodology was studied in classification and regression settings within two real-world application contexts: person recognition using multiple modalities (classification), and head pose estimation using multiple image features (regression). Our experimental results showed that quantile methods of combining p-values (such as the Standard Normal Function and the Non-conformity Aggregation methods) provided the most statistically valid calibration results, and can be considered to extend the CP framework for information fusion settings.

[1]  L. H. C. Tippett The Methods of Statistics. , 1931 .

[2]  M. Kendall Statistical Methods for Research Workers , 1937, Nature.

[3]  A. Gardner Methods of Statistics , 1941 .

[4]  B WILKINSON,et al.  A statistical consideration in psychological research. , 1951, Psychological bulletin.

[5]  J. Hemelrijk,et al.  Some remarks on the combination of independent tests , 1953 .

[6]  Frederick Mosteller,et al.  Selected quantitative techniques and attitude measurement , 1954 .

[7]  I. J. Good,et al.  On the Weighted Combination of Significance Tests , 1955 .

[8]  H. O. Lancaster THE COMBINATION OF PROBABILITIES: AN APPLICATION OF ORTHONORMAL FUNCTIONS , 1961 .

[9]  Eugene S. Edgington,et al.  An Additive Method for Combining Probability Values from Independent Experiments , 1972 .

[10]  Glenn Shafer,et al.  A Mathematical Theory of Evidence , 2020, A Mathematical Theory of Evidence.

[11]  E. Olusegun George,et al.  The Logit Method for Combining Tests. , 1979 .

[12]  Edward H. Shortliffe,et al.  Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence) , 1984 .

[13]  Glenn Shafer,et al.  Perspectives on the theory and practice of belief functions , 1990, Int. J. Approx. Reason..

[14]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 1997, Texts in Computer Science.

[15]  Ming Li,et al.  An Introduction to Kolmogorov Complexity and Its Applications , 2019, Texts in Computer Science.

[16]  Belur V. Dasarathy,et al.  Decision fusion , 1994 .

[17]  Roberto Brunelli,et al.  Person identification using multiple cues , 1995, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  J. Shaffer Multiple Hypothesis Testing , 1995 .

[19]  Stefan Fischer,et al.  Fusion of audio and video information for multi modal person authentication , 1997, Pattern Recognit. Lett..

[20]  Malayappan Shridhar,et al.  Application of fuzzy integrals in fusion of classifiers for low error rate handwritten numerals recognition , 2000, Proceedings 15th International Conference on Pattern Recognition. ICPR-2000.

[21]  Vladimir Vovk,et al.  Ridge Regression Confidence Machine , 2001, International Conference on Machine Learning.

[22]  Norman Poh,et al.  Hybrid Biometric Person Authentication Using Face and Voice Features , 2001, AVBPA.

[23]  F. Pesarin Multivariate Permutation Tests : With Applications in Biostatistics , 2001 .

[24]  Vladimir Vovk,et al.  On-line confidence machines are well-calibrated , 2002, The 43rd Annual IEEE Symposium on Foundations of Computer Science, 2002. Proceedings..

[25]  Harris Papadopoulos,et al.  Inductive Confidence Machines for Regression , 2002, ECML.

[26]  Lisa M. Brown,et al.  Comparative study of coarse head pose estimation , 2002, Workshop on Motion and Video Computing, 2002. Proceedings..

[27]  Alex Park,et al.  MULTI-MODAL FACE AND SPEAKER IDENTIFICATION ON A HANDHELD DEVICE , 2003 .

[28]  Thomas M. Loughin,et al.  A systematic comparison of methods for combining p , 2004, Comput. Stat. Data Anal..

[29]  Galina L. Rogova,et al.  Reliability In Information Fusion : Literature Survey , 2004 .

[30]  Florentin Smarandache,et al.  Advances and Applications of DSmT for Information Fusion (Collected Works) , 2004 .

[31]  Konstantinos Proedrou Rigorous measures of confidence for pattern recognition and regression , 2004 .

[32]  Arun Ross,et al.  Score normalization in multimodal biometric systems , 2005, Pattern Recognit..

[33]  Sethuraman Panchanathan,et al.  A methodology for evaluating robustness of face recognition algorithms with respect to variations in pose angle and illumination angle , 2005, Proceedings. (ICASSP '05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005..

[34]  R.I. Damper,et al.  Fusion of two classifiers for speaker identification: removing and not removing silence , 2005, 2005 7th International Conference on Information Fusion.

[35]  Didier Dubois,et al.  Possibility Theory and its Applications: a Retrospective and Prospective view , 2006, Decision Theory and Multi-Agent Planning.

[36]  Mireia Farrús,et al.  Audio, Video and Multimodal Person Identification in a Smart Room , 2006, CLEAR.

[37]  W. Gasarch,et al.  The Book Review Column 1 Coverage Untyped Systems Simple Types Recursive Types Higher-order Systems General Impression 3 Organization, and Contents of the Book , 2022 .

[38]  Qin Jin,et al.  ISL Person Identification Systems in the CLEAR Evaluations , 2006, CLEAR.

[39]  J.R. Beveridge,et al.  Person Identification Using Text and Image Data , 2007, 2007 First IEEE International Conference on Biometrics: Theory, Applications, and Systems.

[40]  Sethuraman Panchanathan,et al.  Biased Manifold Embedding: A Framework for Person-Independent Head Pose Estimation , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Qin Jin,et al.  Multi-modal Person Identification in a Smart Environment , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[42]  Ralph Gross,et al.  Robust Biometric Person Identification Using Automatic Classifier Fusion of Speech, Mouth, and Face Experts , 2007, IEEE Transactions on Multimedia.

[43]  Domingo Mery,et al.  Bimodal Biometric Person Identification System Under Perturbations , 2007, PSIVT.

[44]  Harris Papadopoulos,et al.  Inductive Conformal Prediction: Theory and Application to Neural Networks , 2008 .

[45]  L. Paola García-Perera,et al.  Enhancing acoustic models for robust speaker verification , 2008, 2008 IEEE International Conference on Acoustics, Speech and Signal Processing.

[46]  Zhiyuan Luo,et al.  Network Traffic Demand Prediction with Confidence , 2008, IEEE GLOBECOM 2008 - 2008 IEEE Global Telecommunications Conference.

[47]  Sethuraman Panchanathan,et al.  Multiple cue integration in transductive confidence machines for head pose classification , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[48]  Arthur P. Dempster,et al.  A Generalization of Bayesian Inference , 1968, Classic Works of the Dempster-Shafer Theory of Belief Functions.

[49]  Eckart Michaelsen,et al.  Evidence fusion using the GESTALT-system , 2008, 2008 11th International Conference on Information Fusion.

[50]  Bayya Yegnanarayana,et al.  Multimodal person authentication using speech, face and visual speech , 2008, Comput. Vis. Image Underst..

[51]  Conrad Sanderson,et al.  Biometric Person Recognition: Face, Speech and Fusion , 2008 .

[52]  Vladimir Vovk,et al.  A tutorial on conformal prediction , 2007, J. Mach. Learn. Res..

[53]  Maurice Milgram,et al.  Boosting feature selection for Neural Network based regression , 2009, Neural Networks.

[54]  Sethuraman Panchanathan,et al.  Generalized Query by Transduction for online active learning , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[55]  Fan Yang,et al.  Using random forest for reliable classification and cost-sensitive learning for medical diagnosis , 2009, BMC Bioinformatics.

[56]  Jorma Laaksonen,et al.  Description, analysis and evaluation of confidence estimation procedures for sub-categorisation , 2009 .

[57]  VN Balasubramanian,et al.  Support vector machine based conformal predictors for risk of complications following a coronary Drug Eluting Stent procedure , 2009, 2009 36th Annual Computers in Cardiology Conference (CinC).

[58]  Mohan M. Trivedi,et al.  Head Pose Estimation in Computer Vision: A Survey , 2009, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Chi-Ho Chan,et al.  On the Results of the First Mobile Biometry (MOBIO) Face and Speaker Verification Evaluation , 2010, ICPR Contests.

[60]  Harris Papadopoulos,et al.  Regression Conformal Prediction with Nearest Neighbours , 2014, J. Artif. Intell. Res..

[61]  Harris Papadopoulos,et al.  Reliable Confidence Measures for Medical Diagnosis With Evolutionary Algorithms , 2011, IEEE Transactions on Information Technology in Biomedicine.