Interaction Challenges in AI Equipped Environments Built to Teach Foreign Languages Through Dialogue and Task-Completion

As cities around the world become more diverse in culture and language, there is a growing need for learning foreign languages. To further this excitement, we have built a human-scale, immersive room with a virtual AI agent that aids foreign language learning. Our system aids the language learning process through task-completion exercises using multi-modal dialogue. The Cognitive and Immersive Room (CIR) is developed as an immersive Chinese restaurant to teach Mandarin, but the interaction challenges and solutions can be reasonably generalized to other languages taught using similar techniques. As users interact with the immersive environment and the virtual AI agent, they face several user interaction challenges. These challenges arise from new learners' lack of proficiency in the foreign language. By studying user interactions in the CIR, we were able to articulate some of the interaction challenges. We have enhanced the AI agent, virtual environment, and the on-boarding process for new users to mitigate these challenges. The enhancements and the results which show that they were effective are discussed here.

[1]  Mau-Tsuen Yang,et al.  The Development of an Online Virtual English Classroom: VEC3D , 2007 .

[2]  Max M. North,et al.  A Comparative Study of Sense of Presence of Traditional Virtual Reality and Immersive Environments , 2016, Australas. J. Inf. Syst..

[3]  Mary Grantham O'Brien,et al.  Virtual Immersion: The Role of CAVE and PC Technology , 2013 .

[4]  Airong Wang.  Learning English in a Multi-User Virtual Environment : Exploring Factors Affecting Participation , 2017 .

[5]  Richard D. Brecht,et al.  Predictors of Foreign Language Gain during Study Abroad. NFLC Occasional Papers. , 1995 .

[6]  Anke Berns,et al.  Game-like language learning in 3-D virtual environments , 2013, Comput. Educ..

[7]  Muna Mohammed Abbas Alkhateeb,et al.  An Analytical Approach of the Cultural Context in Language Teaching and Learning , 2017 .

[8]  Philip Hubbard.  Computer Assisted Language Learning: Critical Concepts in Linguistics , 2009 .

[9]  Wenjing Xu,et al.  Active Learning: Qualitative Inquiries Into Vocabulary Instruction in Chinese L2 Classrooms , 2015 .

[10]  W. Ren.  THE EFFECT OF STUDY ABROAD ON THE PRAGMATIC DEVELOPMENT OF THE INTERNAL MODIFICATION OF REFUSALS , 2013 .

[11]  Xiaofei Tang,et al.  Learning Chinese Formulaic Expressions in a Scenario-Based Interactive Environment , 2017 .

[12]  Qiang Ji,et al.  An Immersive System with Multi-Modal Human-Computer Interaction , 2018, 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018).

[13]  Dan E. Davidson.  Study abroad. When, how long and with what results?: New data from the Russian front , 2010 .

[14]  James Milton,et al.  Foreign language vocabulary development through activities in an online 3D environment , 2012 .

[15]  Lynne Bolduc.  Mastering Difficult Concepts Using Software to Support Learning: A Case Study in Teaching Mandarin Chinese , 1997 .

[16]  David Birchfield,et al.  Earth science learning in SMALLab: A design experiment for mixed reality , 2009, Int. J. Comput. Support. Collab. Learn..

[17]  Aleidine J. Moeller,et al.  Teaching culture in the 21 st century language classroom , 2012 .

[18]  Stephen J. H. Yang,et al.  Designing interaction tasks in Second Life for Chinese as a foreign language learners: A preliminary exploration , 2013 .

[19]  Clare Wright,et al.  Examining the effects of study abroad on L2 Chinese development among UK university learners , 2014 .

[20]  Rong Tong,et al.  Large-scale characterization of non-native Mandarin Chinese spoken by speakers of European origin: Analysis on iCALL , 2016, Speech Commun..

[21]  JEFFREY W. WIDES,et al.  A Preliminary Exploration , 2006 .

[22]  Jiang-Chun Chen,et al.  A prototype of an adaptive Chinese pronunciation training system , 2014 .

[23]  Yu-Ju Lan,et al.  Language Learning in Virtual Reality Environments: Past, Present, and Future , 2015, J. Educ. Technol. Soc..

[24]  Scott Grant,et al.  Language acquisition in Second Life: Improving self-efficacy beliefs , 2009 .

[25]  Silvia Canto,et al.  Language learning effects through the integration of synchronous online communication: The case of video communication and Second Life , 2017 .

[26]  Etsuko Toyoda,et al.  Categorization of Text Chat Communication between Learners and Native Speakers of Japanese. , 2002 .

[27]  Barney Dalgarno,et al.  Scaffolding Discovery Learning In 3D Virtual Environments: Challenges and Considerations for Instructional Design , 2011 .

[28]  Mark Peterson,et al.  Towards a Research Agenda for the Use of Three-Dimensional Virtual Worlds in Language Learning , 2011 .

[29]  Judith Molka-Danielsen,et al.  Designing oral participation in Second Life – a comparative study of two language proficiency courses , 2009, ReCALL.

[30]  Michelle F. Wright,et al.  Computer Gaming and Interactive Simulations for Learning: A Meta-Analysis , 2006 .

[31]  Y. Zhou,et al.  Digital Vocabulary Competition as Motivator for Learning in CFL Classrooms , 2016 .

[32]  Dongdong Chen.  Enhancing the learning of Chinese with Second Life , 2010 .

[33]  Anthony P. Grant,et al.  Effective Language Learning , 1997 .

[34]  H. Creagh.  Cave Automatic Virtual Environment , 2003, Proceedings: Electrical Insulation Conference and Electrical Manufacturing and Coil Winding Technology Conference (Cat. No.03CH37480).

[35]  Roger C. Schank,et al.  Case-Based Teaching: Four Experiences in Educational Software Design , 1990, Interact. Learn. Environ..

[36]  Kasumi Yamazaki,et al.  Computer-assisted learning of communication (CALC): A case study of Japanese learning in a 3D virtual world , 2018, ReCALL.

[37]  Qiang Ji,et al.  CIRA: An Architecture for Building Configurable Immersive Smart-Rooms , 2018, IntelliSys.

[38]  Barbara F. Freed.  Second language acquisition in a study abroad context , 1995 .

[39]  Michael E. Everson.  Best Practices in Teaching Logographic and Non-Roman Writing Systems to L2 Learners , 2011, Annual Review of Applied Linguistics.

[40]  Wenjing Zuo,et al.  The Effects of Cooperative Learning on Improving College Students’ Reading Comprehension , 2011 .

[41]  Selami Aydin,et al.  The Effect of Second Life on Speaking Achievement. , 2016 .

[42]  Yuen-kuang Liao,et al.  The Effect of Computer Simulation Instruction on Student Learning: A Meta-analysis of Studies in Taiwan , 2007 .

[43]  Hang Du,et al.  The Development of Chinese Fluency during Study Abroad in China. , 2013 .

[44]  Ya-Chun Shih,et al.  Weaving Web 2.0 and Facial Expression Recognition into the 3D Virtual English Classroom , 2014 .