Relevance, Effort, and Perceived Quality: Language Learners’ Experiences with AI-Generated Contextually Personalized Learning Material

Artificial intelligence has enabled scalable auto-creation of context-aware personalized learning materials. However, it remains unclear how content personalization shapes the learners’ experience. We developed one personalized and two non-personalized, crowdsourced versions of a mobile language learning app: (1) with personalized auto-generated photo flashcards, (2) the same flashcards provided through crowdsourcing, and (3) manually generated flashcards based on the same photos. A two-week in-situ study (n = 64) showed that learners assessed the quality of the non-personalized auto-generated material to be on par with manually generated material, which means that auto-generation is viable. However, when the auto-generation was personalized, the learners’ quality rating was significantly lower. Further analyses suggest that aspects such as prior expectations and required efforts must be addressed before learners can actually benefit from context-aware personalization with auto-generated material. We discuss design implications and provide an outlook on the role of content personalization in AI-supported learning.

[1]  D. Buschek,et al.  How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Models , 2022, ArXiv.

[2]  F. Babiloni,et al.  Investigating the negative bias towards artificial intelligence: Effects of prior assignment of AI-authorship on the aesthetic appreciation of abstract paintings , 2022, Comput. Hum. Behav..

[3]  L. Chuang,et al.  Agenda- and Activity-Based Triggers for Microlearning , 2022, IUI.

[4]  Niels Pinkwart,et al.  Challenges of using auto-correction tools for language learning , 2022, LAK.

[5]  Joo-Wha Hong,et al.  AI, you can drive my car: How we evaluate human drivers vs. self-driving cars , 2021, Comput. Hum. Behav..

[6]  Qiyao Peng,et al.  Are you ready for artificial Mozart and Skrillex? An experiment testing expectancy violation theory and AI music , 2020, New Media Soc..

[7]  Mei-Rong Alice Chen,et al.  Vocabulary recommendation approach for forced migrants using informal language learning tools , 2021, Universal Access in the Information Society.

[8]  Kartik Talamadupula,et al.  Perfection Not Required? Human-AI Partnerships in Code Translation , 2021, IUI.

[9]  Jan L. Plass,et al.  Toward a taxonomy of adaptivity for learning , 2020, Journal of Research on Technology in Education.

[10]  Candace Walkington,et al.  Appraising research on personalized learning: Definitions, theoretical alignment, advancements, and future directions , 2020 .

[11]  Albrecht Schmidt,et al.  Augmented Reality to Enable Users in Learning Case Grammar from Their Real-World Interactions , 2020, CHI.

[12]  Ting-ting Wu,et al.  Using image-to-text recognition technology to facilitate vocabulary acquisition in authentic contexts , 2020, ReCALL.

[13]  Ben Shneiderman,et al.  Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy , 2020, Int. J. Hum. Comput. Interact..

[14]  Shogo Fukushima,et al.  VocaBura: A Method for Supporting Second Language Vocabulary Learning While Walking , 2019, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[15]  Sangmin-Michelle Lee.  A systematic review of context-aware technology use in foreign language learning , 2019, Computer Assisted Language Learning.

[16]  Hiroaki Ogata,et al.  Vocabulary Learning Support System Based on Automatic Image Captioning Technology , 2019, HCI.

[17]  Carolyn Penstein Rosé,et al.  UpGrade: Sourcing Student Open-Ended Solutions to Create Scalable Learning Opportunities , 2019, L@S.

[18]  Shawn Loewen,et al.  Mobile-assisted language learning: A Duolingo case study , 2019, ReCALL.

[19]  Amy Liu,et al.  Causeway: Scaling Situated Learning with Micro-Role Hierarchies , 2019, CHI.

[20]  Tobias Höllerer,et al.  In-Situ Labeling for Augmented Reality Language Learning , 2019, 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).

[21]  Xiangyu Zhang,et al.  Bounding Box Regression With Uncertainty for Accurate Object Detection , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[22]  A. Graefe,et al.  Readers’ perception of computer-generated news: Credibility, expertise, and readability , 2018 .

[23]  Ming-Ming Cheng,et al.  Review of Visual Saliency Detection With Comprehensive Information , 2018, IEEE Transactions on Circuits and Systems for Video Technology.

[24]  Boualem Benatallah,et al.  Quality Control in Crowdsourcing , 2018, ACM Comput. Surv..

[25]  Matthew L. Bernacki,et al.  Personalization of Instruction: Design Dimensions and Implications for Cognition , 2018 .

[26]  Tobias Höllerer,et al.  ARbis Pictus: A Study of Vocabulary Learning with Augmented Reality , 2017, IEEE Transactions on Visualization and Computer Graphics.

[27]  Albrecht Schmidt,et al.  Building Cognition-Aware Systems , 2017, UbiComp 2017.

[28]  Peter Brusilovsky,et al.  Semi-Supervised Techniques for Mining Learning Outcomes and Prerequisites , 2017, KDD.

[29]  Jeffrey N. Rouder,et al.  Bayesian inference for psychology. Part II: Example applications with JASP , 2017, Psychonomic Bulletin & Review.

[30]  Christian David Vazquez,et al.  Serendipitous Language Learning in Mixed Reality , 2017, CHI Extended Abstracts.

[31]  Solace Shen,et al.  Have your Cake and Eat it Too: Foreign Language Learning with a Crowdsourced Video Captioning System , 2017, CSCW.

[32]  Torrin M. Liddell,et al.  The Bayesian New Statistics: Hypothesis testing, estimation, meta-analysis, and power analysis from a Bayesian perspective , 2017, Psychonomic Bulletin & Review.

[33]  Sari Kujala,et al.  The role of expectations in service evaluation: A longitudinal study of a proximity mobile payment service , 2017, Int. J. Hum. Comput. Stud..

[34]  Joey J. Lee,et al.  A Review of Mobile Language Learning Applications: Trends, Challenges, and Opportunities , 2016 .

[35]  Burr Settles,et al.  A Trainable Spaced Repetition Model for Language Learning , 2016, ACL.

[36]  Neil T. Heffernan,et al.  AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning , 2016, L@S.

[37]  Neil T. Heffernan,et al.  The Future of Adaptive Learning: Does the Crowd Hold the Key? , 2016, International Journal of Artificial Intelligence in Education.

[38]  Jahna Otterbacher,et al.  Crowdsourcing Stereotypes: Linguistic Bias in Metadata Generated via GWAP , 2015, CHI.

[39]  Krzysztof Z. Gajos,et al.  Learnersourcing Subgoal Labels for How-to Videos , 2015, CSCW.

[40]  Elizabeth Ligon Bjork,et al.  Multiple-choice testing as a desirable difficulty in the classroom , 2014 .

[41]  Ching-Kun Hsu,et al.  Learning motivation and adaptive video caption filtering for EFL learners using handheld devices , 2014, ReCALL.

[42]  Hiroaki Ogata,et al.  Ubiquitous Learning Project Using Life-logging Technology in Japan , 2014, J. Educ. Technol. Soc..

[43]  Gwo-Jen Hwang,et al.  Development of an Adaptive Learning System with Multiple Perspectives based on Students? Learning Styles and Cognitive Styles , 2013, J. Educ. Technol. Soc..

[44]  Simon Bates,et al.  Assessing the quality of a student-generated question repository , 2013, 1308.2202.

[45]  Nick A. Hine,et al.  Twasebook: a "crowdsourced phrasebook" for language learners using Twitter , 2012, NordiCHI.

[46]  Thomas Olsson,et al.  Narratives of satisfying and unsatisfying experiences of current mobile augmented reality applications , 2012, CHI.

[47]  J. Mackness,et al.  Connectivism and dimensions of individual experience , 2012 .

[48]  Khai N. Truong,et al.  Augmenting the web for second language vocabulary learning , 2011, CHI.

[49]  Jing Zhao,et al.  MicroMandarin: mobile language learning in context , 2011, CHI.

[50]  Ruth V. Small,et al.  Curiosity, interest and engagement in technology-pervasive learning environments: a new research agenda , 2011 .

[51]  Walt Detmar Meurers,et al.  Enhancing Authentic Web Pages for Language Learners , 2010 .

[52]  Stefaan Ternier,et al.  Mobile Learning in Context , 2010 .

[53]  Kathryn R. Wentzel,et al.  Promoting Self-Determined School Engagement: Motivation, Learning, and Well-Being , 2009 .

[54]  Ryen W. White,et al.  Exploratory Search: Beyond the Query-Response Paradigm , 2009, Exploratory Search: Beyond the Query-Response Paradigm.

[55]  Gwo-Jen Hwang,et al.  Criteria, Strategies and Research Issues of Context-Aware Ubiquitous Learning , 2008, J. Educ. Technol. Soc..

[56]  Margaret E. Morris,et al.  Context-Sensitive Microlearning of Foreign Language Vocabulary on a Mobile Device , 2007, AmI.

[57]  Yvonne Rogers,et al.  Why It's Worth the Hassle: The Value of In-Situ Studies When Designing Ubicomp , 2007, UbiComp.

[58]  S. Hidi,et al.  The Four-Phase Model of Interest Development , 2006 .

[59]  J. B. Brooke,et al.  SUS: A 'Quick and Dirty' Usability Scale , 1996 .

[60]  J. Burgoon.  Interpersonal Expectations, Expectancy Violations, and Emotional Communication , 1993 .

[61]  Albrecht Schmidt,et al.  An Environment-Triggered Augmented-Reality Application for Learning Case Grammar , 2020, DeLFI.

[62]  Albrecht Schmidt,et al.  Building Cognition-Aware Systems: A Mobile Toolkit for Extracting Time-of-Day Fluctuations of Cognitive Performance , 2017, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol..

[63]  Ullrich K. H. Ecker,et al.  Misinformation and Its Correction , 2012, Psychological science in the public interest : a journal of the American Psychological Society.

[64]  E. Deci,et al.  Promoting self-determined school engagement: Motivation, learning, and well-being. , 2009 .

[65]  Ryen W. White,et al.  Exploratory Search , 2008 .

[66]  M. Malbrán.  The Cambridge Handbook of Multimedia Learning , 2007 .