Visual Knowledge Tracing

. Each year, thousands of people learn new visual categorization tasks – radiologists learn to recognize tumors, birdwatchers learn to distinguish similar species, and crowd workers learn how to annotate valuable data for applications like autonomous driving. As humans learn, their brain updates the visual features it extracts and attend to, which ultimately informs their final classification decisions. In this work, we propose a novel task of tracing the evolving classification behavior of human learners as they engage in challenging visual classification tasks. We propose models that jointly extract the visual features used by learners as well as predicting the classification functions they utilize. We collect three challenging new datasets from real human learners in order to evaluate the performance of different visual knowledge tracing methods. Our results show that our recurrent models are able to predict the classification behavior of human learners on three challenging medical image and species identification tasks.

[1]  Oisin Mac Aodha,et al.  When Does Contrastive Visual Representation Learning Work? , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Daniel N. Barry,et al.  Human learning follows the dynamics of gradient descent , 2021 .

[3]  Pei Wang,et al.  A Machine Teaching Framework for Scalable Recognition , 2021, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[4]  N. Vasconcelos,et al.  Gradient-based Algorithms for Machine Teaching , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Enhong Chen,et al.  A Survey of Knowledge Tracing , 2021, ArXiv.

[6]  Brett D. Roads.  Predicting the Ease of Human Category Learning Using Radial Basis Function Networks , 2021, Neural Computation.

[7]  Bradley C. Love,et al.  Enriching ImageNet with Human Similarity Judgments and Psychological Embeddings , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  R. Nosofsky,et al.  Contrasting Exemplar and Prototype Models in a Natural-Science Category Domain , 2022, CogSci.

[9]  Evan M. Palmer,et al.  Training Novices to Discriminate Retinal Diseases Using Perceptual Learning , 2020, Proceedings of the Human Factors and Ergonomics Society Annual Meeting.

[10]  C. Chaou,et al.  Artificial intelligence-based education assists medical students’ interpretation of hip fracture , 2020, Insights into Imaging.

[11]  Michael C. Mozer,et al.  Transforming Neural Network Visual Representations to Predict Human Judgments of Similarity , 2020, ArXiv.

[12]  Michael Yudelson,et al.  Deep Knowledge Tracing with Transformers , 2020, AIED.

[13]  Charles Y. Zheng,et al.  Revealing the multidimensional mental representations of natural objects underlying human similarity judgments , 2020, Nature Human Behaviour.

[14]  Ser-Nam Lim,et al.  A Metric Learning Reality Check , 2020, ECCV.

[15]  Joseph Paul Cohen,et al.  Revisiting Training Strategies and Generalization Performance in Deep Metric Learning , 2020, ICML.

[16]  Byungsoo Kim,et al.  Towards an Appropriate Query, Key, and Value Computation for Knowledge Tracing , 2020, L@S.

[17]  Geoffrey E. Hinton,et al.  A Simple Framework for Contrastive Learning of Visual Representations , 2020, ICML.

[18]  Ross B. Girshick,et al.  Momentum Contrast for Unsupervised Visual Representation Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[19]  J. Uijlings,et al.  The Open Images Dataset V4 , 2018, International Journal of Computer Vision.

[20]  Deva Ramanan,et al.  Towards Latent Attribute Discovery From Triplet Similarities , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[21]  Kate Saenko,et al.  Learning Similarity Conditions Without Explicit Supervision , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[22]  Hasan Şakir Bilge,et al.  Deep Metric Learning: A Survey , 2019, Symmetry.

[23]  George Karypis,et al.  A Self Attentive model for Knowledge Tracing , 2019, EDM.

[24]  Andreas Krause,et al.  Teaching Multiple Concepts to a Forgetful Learner , 2018, NeurIPS.

[25]  Robert M Nosofsky,et al.  Toward the development of a feature-space representation for a complex natural category domain , 2017, Behavior Research Methods.

[26]  Daniel S. Kermany,et al.  Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning , 2018, Cell.

[27]  Pietro Perona,et al.  Teaching Categories to Human Learners with Visual Explanations , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[28]  Sandra Zilles,et al.  An Overview of Machine Teaching , 2018, ArXiv.

[29]  Pietro Perona,et al.  Context Embedding Networks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[30]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[31]  Serge J. Belongie,et al.  Conditional Similarity Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Burr Settles,et al.  A Trainable Spaced Repetition Model for Language Learning , 2016, ACL.

[33]  Leonidas J. Guibas,et al.  Deep Knowledge Tracing , 2015, NIPS.

[34]  Pietro Perona,et al.  Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[35]  Gabriel J. Brostow,et al.  Becoming the expert - interactive multi-class machine teaching , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[36]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[38]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[39]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[40]  Yang Song,et al.  Learning Fine-Grained Image Similarity with Deep Ranking , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[41]  Serge J. Belongie,et al.  Cost-Effective HITs for Relative Similarity Comparisons , 2014, HCOMP.

[42]  Andreas Krause,et al.  Near-Optimally Teaching the Crowd to Classify , 2014, ICML.

[43]  Michael C. Hout,et al.  Multidimensional Scaling , 2003, Encyclopedic Dictionary of Archaeology.

[44]  Kilian Q. Weinberger,et al.  Stochastic triplet embedding , 2012, 2012 IEEE International Workshop on Machine Learning for Signal Processing.

[45]  Adam Tauman Kalai,et al.  Adaptively Learning the Crowd Kernel , 2011, ICML.

[46]  W. T. Maddox,et al.  Annals of the New York Academy of Sciences Human Category Learning 2.0 Brief Review of First-generation Research , 2022 .

[47]  Pietro Perona,et al.  The Multidimensional Wisdom of Crowds , 2010, NIPS.

[48]  Dagmar Zeithamova.  Category Learning Systems , 2008 .

[49]  J. Tenenbaum,et al.  Word learning as Bayesian inference. , 2007, Psychological review.

[50]  F. Ashby,et al.  Category learning and multiple memory systems , 2005, Trends in Cognitive Sciences.

[51]  John R. Anderson,et al.  Knowledge tracing: Modeling the acquisition of procedural knowledge , 2005, User Modeling and User-Adapted Interaction.

[52]  David L. Faigman,et al.  Human category learning. , 2005, Annual review of psychology.

[53]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[54]  Ellen M. Markman,et al.  Categorization and Naming in Children: Problems of Induction , 1989 .

[55]  I. Biederman,et al.  Sexing Day-Old Chicks : A Case Study and Expert Systems Analysis of a Difficult Perceptual-Learning Task , 1987 .