Employing UMLS for generating hints in a tutoring system for medical problem-based learning

While problem-based learning has become widely popular for imparting clinical reasoning skills, the dynamics of medical PBL require close attention to a small group of students, placing a burden on medical faculty, whose time is over taxed. Intelligent tutoring systems (ITSs) offer an attractive means to increase the amount of facilitated PBL training the students receive. But typical intelligent tutoring system architectures make use of a domain model that provides a limited set of approved solutions to problems presented to students. Student solutions that do not match the approved ones, but are otherwise partially correct, receive little acknowledgement as feedback, stifling broader reasoning. Allowing students to creatively explore the space of possible solutions is exactly one of the attractive features of PBL. This paper provides an alternative to the traditional ITS architecture by using a hint generation strategy that leverages a domain ontology to provide effective feedback. The concept hierarchy and co-occurrence between concepts in the domain ontology are drawn upon to ascertain partial correctness of a solution and guide student reasoning towards a correct solution. We describe the strategy incorporated in METEOR, a tutoring system for medical PBL, wherein the widely available UMLS is deployed and represented as the domain ontology. Evaluation of expert agreement with system generated hints on a 5-point likert scale resulted in an average score of 4.44 (Spearman's ρ=0.80, p<0.01). Hints containing partial correctness feedback scored significantly higher than those without it (Mann Whitney, p<0.001). Hints produced by a human expert received an average score of 4.2 (Spearman's ρ=0.80, p<0.01).

[1]  Ted Pedersen,et al.  Measures of semantic similarity and relatedness in the biomedical domain , 2007, J. Biomed. Informatics.

[2]  Antonija Mitrovic,et al.  A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling , 2003, User Modeling.

[3]  John R. Anderson,et al.  Cognitive Tutors: Lessons Learned , 1995 .

[4]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[5]  Siriwan Suebnukarn,et al.  Expanding the Space of Plausible Solutions in a Medical Tutoring System for Problem-Based Learning , 2009, Int. J. Artif. Intell. Educ..

[6]  Siriwan Suebnukarn,et al.  A Bayesian approach to generating tutorial hints in a collaborative medical problem-based learning system , 2006, Artif. Intell. Medicine.

[7]  John Self.  Bypassing the intractable problem of student modelling , 1988 .

[8]  Torsten Illmann,et al.  Docs 'n drugs-the virtual polyclinic: an intelligent tutoring system for web-based and case-oriented training in medicine , 2001, AMIA.

[9]  Antonija Mitrovic,et al.  A Knowledge Acquisition System for Constraint-based Intelligent Tutoring Systems , 2005, AIED.

[10]  Rebecca S. Crowley,et al.  ReportTutor - An Intelligent Tutoring System that Uses a Natural Language Interface , 2005, AMIA.

[11]  Hoa A. Nguyen,et al.  A Cluster-Based Approach for Semantic Similarity in the Biomedical Domain , 2006, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[12]  Kurt VanLehn,et al.  The Andes Physics Tutoring System: Lessons Learned , 2005, Int. J. Artif. Intell. Educ..

[13]  Siriwan Suebnukarn,et al.  Expanding the Plausible Solution Space for Robustness in an Intelligent Tutoring System , 2008, Intelligent Tutoring Systems.

[14]  Siriwan Suebnukarn,et al.  A collaborative medical case authoring environment based on the UMLS , 2008, J. Biomed. Informatics.

[15]  David Rosenthal,et al.  An Assessment of Constraint-Based Tutors: A Response to Mitrovic and Ohlsson's Critique of "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms" , 2006, Int. J. Artif. Intell. Educ..

[16]  Antonija Mitrovic,et al.  Experiences in Implementing Constraint-Based Modeling in SQL-Tutor , 1998, Intelligent Tutoring Systems.

[17]  Guey-Fa Chiou,et al.  Designing an ontology-based intelligent tutoring agent with instant messaging , 2005, Fifth IEEE International Conference on Advanced Learning Technologies (ICALT'05).

[18]  Christiane Fellbaum,et al.  Combining Local Context and Wordnet Similarity for Word Sense Identification , 1998 .

[19]  Hisham Al-Mubaid,et al.  Using MEDLINE as Standard Corpus for Measuring Semantic Similarity in the Biomedical Domain , 2006, Sixth IEEE Symposium on BioInformatics and BioEngineering (BIBE'06).

[20]  Reva Freedman,et al.  Implementing directed lines of reasoning in an intelligent tutoring system using the atlas planning environment , 2004, International Conference on Information Technology: Coding and Computing, 2004. Proceedings. ITCC 2004..

[21]  James J. Cimino,et al.  Automated knowledge extraction from MEDLINE citations , 2000, AMIA.

[22]  A. Mitrovic,et al.  Easing the ITS Bottleneck with Constraint-Based Modelling , 2022 .

[23]  Siriwan Suebnukarn,et al.  Leveraging a Domain Ontology to Increase the Quality of Feedback in an Intelligent Tutoring System , 2010, Intelligent Tutoring Systems.

[24]  Philip Resnik,et al.  Semantic Similarity in a Taxonomy: An Information-Based Measure and its Application to Problems of Ambiguity in Natural Language , 1999, J. Artif. Intell. Res..

[25]  Rebecca S. Crowley,et al.  An intelligent tutoring system for visual classification problem solving , 2006, Artif. Intell. Medicine.

[26]  Siriwan Suebnukarn,et al.  Modeling individual and collaborative problem-solving in medical problem-based learning , 2006, User Modeling and User-Adapted Interaction.

[27]  James J. Cimino,et al.  Automated knowledge extraction from the UMLS , 1998, AMIA.

[28]  Dimitra Tsovaltzi,et al.  Domain-Knowledge Manipulation for Dialogue-Adaptive Hinting , 2005, AIED.

[29]  Tom Mu.  Expanding the Knowledge Acquisition Bottleneck for Intelligent Tutoring Systems , 2009 .

[30]  H S Barrows,et al.  A taxonomy of problem‐based learning methods , 1986, Medical education.

[31]  James J. Cimino,et al.  Towards the development of a conceptual distance metric for the UMLS , 2004, J. Biomed. Informatics.

[32]  David Sánchez,et al.  An ontology-based measure to compute semantic similarity in biomedicine , 2011, J. Biomed. Informatics.

[33]  Roy Rada,et al.  Development and application of a metric on semantic nets , 1989, IEEE Trans. Syst. Man Cybern..

[34]  Michel Dojat,et al.  Application of Information Technology: A UMLS-based Knowledge Acquisition Tool for Rule-based Clinical Decision Support System Development , 2001, J. Am. Medical Informatics Assoc..