Experimental Design under the Bradley-Terry Model

Labels generated by human experts via comparisons exhibit smaller variance compared to traditional sample labels. Collecting comparison labels is challenging over large datasets, as the number of comparisons grows quadratically with the dataset size. We study the following experimental design problem: given a budget of expert comparisons, and a set of existing sample labels, we determine the comparison labels to collect that lead to the highest classification improvement. We study several experimental design objectives motivated by the BradleyTerry model. The resulting optimization problems amount to maximizing submodular functions. We experimentally evaluate the performance of these methods over synthetic and real-life datasets.

[1]  Rishabh K. Iyer,et al.  Submodularity in Data Subset Selection and Active Learning , 2015, ICML.

[2]  K. Chaloner,et al.  Bayesian Experimental Design: A Review , 1995 .

[3]  Thomas G. Dietterich.  What is machine learning? , 2020, Archives of Disease in Childhood.

[4]  Manali Sharma,et al.  Evidence-based uncertainty sampling for active learning , 2016, Data Mining and Knowledge Discovery.

[5]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..

[6]  Rong Jin,et al.  Batch mode active learning and its application to medical image classification , 2006, ICML.

[7]  Michael I. Jordan,et al.  Robust design of biological experiments , 2005, NIPS.

[8]  Roger B. Myerson,et al.  Optimal Auction Design , 1981, Math. Oper. Res..

[9]  D. Sculley,et al.  Combined regression and ranking , 2010, KDD.

[10]  David A. Cohn,et al.  Active Learning with Statistical Models , 1996, NIPS.

[11]  Stratis Ioannidis,et al.  Budget Feasible Mechanisms for Experimental Design , 2013, LATIN.

[12]  Jeff A. Bilmes,et al.  Active Semi-Supervised Learning using Submodular Functions , 2011, UAI.

[13]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[14]  Anthony N Pettitt,et al.  Bayesian Experimental Design for Models with Intractable Likelihoods , 2013, Biometrics.

[15]  Grey Giddins,et al.  Statistics , 2016, The Journal of hand surgery, European volume.

[16]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS THE METHOD OF PAIRED COMPARISONS , 1952 .

[17]  Heinz Holling,et al.  Optimal paired comparison designs for first-order interactions , 2003 .

[18]  A. Elo.  The rating of chessplayers, past and present , 1978 .

[19]  Tong Zhang,et al.  The Value of Unlabeled Data for Classification Problems , 2000, ICML 2000.

[20]  Hisashi Kashima,et al.  Progressive Comparison for Ranking Estimation , 2016, IJCAI.

[21]  Rainer Schwabe,et al.  Optimal design for the Bradley–Terry paired comparison model , 2008, Stat. Methods Appl..

[22]  C. R. Rao,et al.  Information and the Accuracy Attainable in the Estimation of Statistical Parameters , 1992 .

[23]  D. Harville.  Matrix Algebra From a Statistician's Perspective , 1998 .

[24]  Andreas Krause,et al.  Optimal Nonmyopic Value of Information in Graphical Models - Efficient Algorithms and Theoretical Limits , 2005, IJCAI.

[25]  Michael P. H. Stumpf,et al.  Maximizing the Information Content of Experiments in Systems Biology , 2013, PLoS Comput. Biol..

[26]  William A. Gale,et al.  A sequential algorithm for training text classifiers , 1994, SIGIR '94.

[27]  Deniz Erdogmus,et al.  Plus Disease in Retinopathy of Prematurity: Improving Diagnosis by Ranking Disease Severity and Using Quantitative Image Analysis. , 2016, Ophthalmology.

[28]  R. A. Bradley,et al.  RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS , 1952 .

[29]  Roger G. Ghanem,et al.  Efficient Bayesian Experimentation Using an Expected Information Gain Lower Bound , 2015, SIAM/ASA J. Uncertain. Quantification.

[30]  Shotaro Akaho,et al.  TrBagg: A Simple Transfer Learning Method and its Application to Personalization in Collaborative Tagging , 2009, 2009 Ninth IEEE International Conference on Data Mining.

[31]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[32]  Anthony N. Pettitt,et al.  A Sequential Monte Carlo Algorithm to Incorporate Model Uncertainty in Bayesian Sequential Design , 2014 .

[33]  K. J. Ryan,et al.  Estimating Expected Information Gains for Experimental Designs With Application to the Random Fatigue-Limit Model , 2003 .

[34]  Shane T. Jensen,et al.  Adaptive Paired Comparison Design , 2005 .

[35]  Michael F Chiang,et al.  Agreement among pediatric ophthalmologists in diagnosing plus and pre-plus disease in retinopathy of prematurity. , 2008, Journal of AAPOS : the official publication of the American Association for Pediatric Ophthalmology and Strabismus.

[36]  Xun Huan,et al.  Simulation-based optimal Bayesian experimental design for nonlinear systems , 2011, J. Comput. Phys..

[37]  Mark A. Pitt,et al.  Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science , 2010, Neural Computation.

[38]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[39]  Baoxin Li,et al.  Fusing Pointwise and Pairwise Labels for Supporting User-adaptive Image Retrieval , 2015, ICMR.

[40]  Baoxin Li,et al.  PPP: Joint Pointwise and Pairwise Image Label Prediction , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).