Uncertainty Representation in Visualizations of Learning Analytics for Learners: Current Approaches and Opportunities

Adding uncertainty information to visualizations is becoming increasingly common across domains since its addition helps ensure that informed decisions are made. This work has shown the difficulty that is inherent to representing uncertainty. Moreover, the representation of uncertainty has yet to be thoroughly explored in educational domains even though visualizations are often used in educational reporting. We analyzed 50 uncertainty-augmented visualizations from various disciplines to map out how uncertainty has been represented. We then analyzed 106 visualizations from educational reporting systems where the learner can see the visualization; these visualizations provide learners with information about several factors including their knowledge, performance, and abilities. This analysis mapped the design space that has been employed to communicate a learner's abilities, knowledge, and interests. It also revealed several opportunities for the inclusion of uncertainty information within visualizations of educational data. We describe how uncertainty information can be added to visualizations of educational data and illustrate these opportunities by augmenting several of the types of visualizations that are found in existing learning analytics reports. The definition of this design space, based on a survey of the literature, will enable the systematic exploration of how different design decisions affect learner trust, understanding, and decision making.

[1]  David Rosenthal,et al.  An Assessment of Constraint-Based Tutors: A Response to Mitrovic and Ohlsson's Critique of "A Comparison of Model-Tracing and Constraint-Based Intelligent Tutoring Paradigms" , 2006, Int. J. Artif. Intell. Educ..

[2]  Stacy Marsella,et al.  The DARWARS Tactical Language Training System , 2004 .

[3]  Susan Bull,et al.  Formative Assessment and Meaningful Learning Analytics , 2014, 2014 IEEE 14th International Conference on Advanced Learning Technologies.

[4]  Paul Brna,et al.  Opening up the Interpretation Process in an Open Learner Model , 2007, Int. J. Artif. Intell. Educ..

[5]  Kwan-Liu Ma,et al.  A framework for uncertainty-aware visual analytics , 2009, 2009 IEEE Symposium on Visual Analytics Science and Technology.

[6]  Susan Bull,et al.  Visualising Uncertainty for Open Learner Model Users , 2014, UMAP Workshops.

[7]  G. McCalla,et al.  Active Open Learner Modelling , 2003 .

[8]  Richard A. Becker,et al.  Brushing scatterplots , 1987 .

[9]  Mark Gahegan,et al.  Visual Semiotics & Uncertainty Visualization: An Empirical Study , 2012, IEEE Transactions on Visualization and Computer Graphics.

[10]  Torre Zuk,et al.  Visualizing uncertainty , 2008 .

[11]  Zhi-Hong Chen,et al.  Active Open Learner Models as Animal Companions: Motivating Children to Learn through Interacting with My-Pet and Our-Pet , 2007, Int. J. Artif. Intell. Educ..

[12]  Michael Gleicher,et al.  Error Bars Considered Harmful: Exploring Alternate Encodings for Mean and Error , 2014, IEEE Transactions on Visualization and Computer Graphics.

[13]  Gerhard Weber,et al.  Episodic Learner Modeling , 1996, Cogn. Sci..

[14]  Jack Burston.  Review of BonPatron: An Online Spelling, Grammar, and Expression Checker , 2013 .

[15]  P. Fisher.  Visualizing Uncertainty in Soil Maps by Animation , 1993 .

[16]  Bert Reynvoet,et al.  The Role of Visual Information in Numerosity Estimation , 2012, PloS one.

[17]  John Self.  Bypassing the intractable problem of student modelling , 1988 .

[18]  Tobias Isenberg,et al.  Evaluating Sketchiness as a Visual Variable for the Depiction of Qualitative Uncertainty , 2012, IEEE Transactions on Visualization and Computer Graphics.

[19]  Mary Hegarty,et al.  Comparing Graphical and Verbal Representations of Measurement Error In Test Score Reports , 2014 .

[20]  Antonija Mitrovic,et al.  Assessing the Impact of Positive Feedback in Constraint-Based Tutors , 2008, Intelligent Tutoring Systems.

[21]  Jing Zhao,et al.  MicroMandarin: mobile language learning in context , 2011, CHI.

[22]  Julita Vassileva,et al.  Evolving a Social Visualization Design Aimed at Increasing Participation in a Class-Based Online Community , 2008, Int. J. Cooperative Inf. Syst..

[23]  Susan Bull,et al.  Student Preferences for Editing, Persuading, and Negotiating the Open Learner Model , 2006, Intelligent Tutoring Systems.

[24]  Ann M. Bisantz,et al.  Comparing Uncertainty Visualizations for a Dynamic Decision-Making Task , 2011 .

[25]  Erik Duval,et al.  Visualizing Activities for Self-reflection and Awareness , 2010, ICWL.

[26]  Carl Gutwin,et al.  Seeing things in the clouds: the effect of visual features on tag cloud selections , 2008, Hypertext.

[27]  Brian Falkenhainer,et al.  The Structure-Mapping Engine: Algorithm and Examples , 1989, Artif. Intell..

[28]  Susan Bull,et al.  Group Interaction Prompted by a Simple Assessed Open Learner Model that can be Optionally Released to Peers , 2007 .

[29]  Susan Bull,et al.  A haptic learner model , 2006 .

[30]  Moffat Mathews.  Investigating the Effectiveness of Problem Templates on Learning in Intelligent Tutoring Systems BSc Honours Report , 2006 .

[31]  Birgitta König-Ries,et al.  Open social student modeling: visualizing student models with parallel introspectiveviews , 2011, UMAP'11.

[32]  G. Cumming,et al.  Researchers misunderstand confidence intervals and standard error bars. , 2005, Psychological methods.

[33]  Susan Bull,et al.  An Intelligent Learning Environment with an Open Learner Model for the Desktop PC and Pocket PC , 2003 .

[34]  Colin Ware,et al.  Information Visualization: Perception for Design , 2000 .

[35]  Judy Kay,et al.  Exploiting Readily Available Web Data for Scrutable Student Models , 2005, AIED.

[36]  Lucy Bastin,et al.  Visualizing uncertainty in multi-spectral remotely sensed imagery , 2002 .

[37]  Paul Rosen,et al.  From Quantification to Visualization: A Taxonomy of Uncertainty Visualization Approaches , 2011, WoCoUQ.

[38]  K. VanLehn.  The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems , 2011 .

[39]  Judy Kay,et al.  Student Models that Invite the Learner In: The SMILI: () Open Learner Modelling Framework , 2007, Int. J. Artif. Intell. Educ..

[40]  Robert M. Edsall,et al.  The Influence of Uncertainty Visualization on Decision Making: An Empirical Evaluation , 2006 .

[41]  Jim E. Greer,et al.  Interacting with Inspectable Bayesian Student Models , 2004, Int. J. Artif. Intell. Educ..

[42]  Alyssa Friend Wise,et al.  Designing pedagogical interventions to support student use of learning analytics , 2014, LAK.

[43]  Thenkurussi Kesavadas,et al.  The Effect of Display Modality on Decision-Making with Uncertainty , 2003 .

[44]  Ann M. Bisantz,et al.  Displaying Uncertainty: Investigating the Effects of Display Format and Specificity , 2005, Hum. Factors.

[45]  Ganesh S. Oak.  Information Visualization Introduction , 2022 .

[46]  William J. Byrne,et al.  Visualising Multiple Data Sources in an Independent Open Learner Model , 2013, AIED.

[47]  Judy Kay,et al.  Creating personalized systems that people can scrutinize and control: Drivers, principles and experience , 2012, TIIS.

[48]  Nahum D. Gershon.  Visualization of an Imperfect World , 1998, IEEE Computer Graphics and Applications.

[49]  Chris R. Johnson,et al.  A Next Step: Visualizing Errors and Uncertainty , 2003, IEEE Computer Graphics and Applications.

[50]  E. Duval.  Attention please!: learning analytics for visualization and recommendation , 2011, LAK.

[51]  Gordon I. McCalla,et al.  ProTutor: Historic Open Learner Models for Pronunciation Tutoring , 2011, AIED.

[52]  Antonija Mitrovic,et al.  A Comparative Analysis of Cognitive Tutoring and Constraint-Based Modeling , 2003, User Modeling.

[53]  Judy Kay,et al.  Narcissus: Group and Individual Models to Support Small Group Work , 2009, UMAP.

[54]  M. Sheelagh T. Carpendale,et al.  Visualization of Uncertainty in Lattices to Support Decision-Making , 2007, EuroVis.

[55]  Brian J Scholl,et al.  Bar graphs depicting averages are perceptually misinterpreted: The within-the-bar bias , 2012, Psychonomic bulletin & review.

[56]  Ken Brodlie,et al.  A Review of Uncertainty in Data Visualization , 2012, Expanding the Frontiers of Visual Analytics and Visualization.

[57]  Alex T. Pang,et al.  UFLOW: visualizing uncertainty in fluid flow , 1996, Proceedings of Seventh Annual IEEE Visualization '96.

[58]  Konstantina Chrysafiadi,et al.  Student modeling approaches: A literature review for the last decade , 2013, Expert Syst. Appl..

[59]  Carolyn Penstein Rosé,et al.  Fading and Deepening: The Next Steps for Andes and other Model-Tracing Tutors , 2000, Intelligent Tutoring Systems.

[60]  Carl Gutwin,et al.  Useful junk?: the effects of visual embellishment on comprehension and memorability of charts , 2010, CHI.

[61]  Susan Bull,et al.  Do Students Trust Their Open Learner Models? , 2008, AH.

[62]  S. Blue,et al.  Visual complexity in infancy: contour or numerosity? , 1975, Child development.

[63]  Ryan S. Baker,et al.  The State of Educational Data Mining in 2009: A Review and Future Visions. , 2009, EDM 2009.

[64]  Mark Gahegan,et al.  A typology for visualizing uncertainty , 2005, IS&T/SPIE Electronic Imaging.

[65]  Susan Bull,et al.  Learner Trust in Learner Model Externalisations , 2009, AIED.

[66]  Antonija Mitrovic,et al.  Evaluating the Effect of Open Student Models on Self-Assessment , 2007, Int. J. Artif. Intell. Educ..

[67]  Judith Masthoff,et al.  Layered evaluation of interactive adaptive systems: framework and formative methods , 2010, User Modeling and User-Adapted Interaction.

[68]  D. Vaux,et al.  Error bars in experimental biology , 2007, The Journal of cell biology.

[69]  Seppo Pohjolainen,et al.  Students’ Motivations for Social Media Enhanced Studying and Learning , 2010 .

[70]  Edward Rolf Tufte,et al.  The visual display of quantitative information , 1985 .

[71]  Tobias Isenberg,et al.  Sketchy Rendering for Information Visualization , 2012, IEEE Transactions on Visualization and Computer Graphics.

[72]  Albert T. Corbett,et al.  The Impact on Learning of Generating vs. Selecting Descriptions in Analyzing Algebra Example Solutions , 2006, ICLS.

[73]  Mike Pearson,et al.  Visualizing Uncertainty About the Future , 2022 .

[74]  Bongshin Lee,et al.  Revealing Uncertainty for Information Visualization , 2010, Inf. Vis..

[75]  Stellan Ohlsson,et al.  Constraint-Based Student Modeling , 1994 .

[76]  Alex T. Pang,et al.  Glyphs for Visualizing Uncertainty in Vector Fields , 1996, IEEE Trans. Vis. Comput. Graph..

[77]  John J. Bertin,et al.  The semiology of graphics , 1983 .

[78]  Bert Bredeweg,et al.  Student Modelling: The Key to Individualized Knowledge-Based Instruction , 2010, NATO ASI Series.

[79]  H. Pain,et al.  ' Did I say what I think I said , and do you agree with me ? ' : Inspecting and Questioning the Student Model , 1995 .

[80]  James Uther,et al.  On the Visualisation of Large User Models in Web Based Systems , 2001 .

[81]  Gord McCalla,et al.  The Ecological Approach to the Design of E-Learning Environments: Purpose-based Capture and Use of Information About Learners , 2004 .

[82]  Julie Maitland,et al.  ALEX: mobile language assistant for low-literacy adults , 2010, Mobile HCI.

[83]  P. Fayers,et al.  The Visual Display of Quantitative Information , 1990 .

[84]  S. Bull,et al.  Reactions to Inspectable Learner Models : Seven Year Olds to University Students , 2005 .

[85]  Erik Duval,et al.  Learning Dashboards & Learnscapes , 2012, CHI 2012.

[86]  Jim Hewitt,et al.  A Study of Private Messaging Within an Asynchronous Discussion Environment , 2013, CSCL.