Integrating Affective Responses and Gamification into Early Reading Acquisition Software Applications

Sisu is a gamified learning application designed to assist school-aged children who are struggling to read. Sisu utilizes readily-available technology to promote learning at home, with unique elements tied to the learning experience: (1) a spelling game with (2) an empathic agent, and (3) a mini-game. The empathic agent utilizes a facial action coding system (FACS) to recognize core expressions of the child user and respond to the child's affect in-game. We anticipate that Sisu's accessible and affective nature will not only support children's emotional needs, but the addition of gamified elements will motivate them to practice reading and assist them in their learning objectives.

[1]  Jeanne Wanzek,et al.  Response to Intervention. , 2015, European scientific journal.

[2]  Jon A. Hess,et al.  Sustaining the desire to learn: Dimensions of perceived instructional facework related to student involvement and motivation to learn , 2003 .

[3]  Rosalind W. Picard.  Affective computing: (526112012-054) , 1997 .

[4]  J. Ziegler,et al.  Extra-large letter spacing improves reading in dyslexia , 2012, Proceedings of the National Academy of Sciences.

[5]  L. Ehri.  Learning to Read and Spell Words , 1987 .

[6]  Rebeca P. Díaz Redondo,et al.  An Experiment to Assess Students’ Engagement in a Gamified Social Learning Environment , 2015 .

[7]  Ricardo Baeza-Yates,et al.  Good fonts for dyslexia , 2013, ASSETS.

[8]  Tove Gerholm,et al.  Children's development of facework practicesAn emotional endeavor , 2011 .

[9]  Hae Won Park,et al.  Flat vs. Expressive Storytelling: Young Children’s Learning and Retention of a Social Robot’s Narrative , 2017, Front. Hum. Neurosci..

[10]  P. Ekman,et al.  What the face reveals : basic and applied studies of spontaneous expression using the facial action coding system (FACS) , 2005 .

[11]  Marianne Martens,et al.  Reading and “Gamification”: Joining Guilds, Earning Badges, and Leveling Up , 2015 .

[12]  Mitsuru Ishizuka,et al.  THE EMPATHIC COMPANION: A CHARACTER-BASED INTERFACE THAT ADDRESSES USERS' AFFECTIVE STATES , 2005, Appl. Artif. Intell..

[13]  Po-Yao Chao,et al.  An Empathic Avatar in a Computer-Aided Learning Program to Encourage and Persuade Learners , 2012, J. Educ. Technol. Soc..

[14]  Patricia F. Vadasy,et al.  Effects of Tutoring in Phonological and Early Reading Skills on Students at Risk for Reading Disabilities , 2000, Journal of learning disabilities.

[15]  Tamás D. Gedeon,et al.  Emotion Recognition In The Wild Challenge 2014: Baseline, Data and Protocol , 2014, ICMI.

[16]  G. Clore,et al.  Mood and the use of scripts: does a happy mood really lead to mindlessness? , 1996, Journal of personality and social psychology.

[17]  Linnea C. Ehri,et al.  Orthographic Mapping in the Acquisition of Sight Word Reading, Spelling Memory, and Vocabulary Learning , 2014 .

[18]  Ronald D. Davis,et al.  The Gift of Dyslexia: Why Some of the Brightest People Can't Read and How They Can Learn , 1993 .

[19]  Ashish Kapoor,et al.  Multimodal affect recognition in learning environments , 2005, ACM Multimedia.

[20]  Ivica Boticki,et al.  Usage of a mobile social learning platform with virtual badges in a primary school , 2015, Comput. Educ..

[21]  Cynthia Breazeal,et al.  A study to measure the effect of framing a robot as a social agent or as a machine on children's social behavior , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[22]  Julie A. Kientz,et al.  Screen Time Tantrums: How Families Manage Screen Media Experiences for Toddlers and Preschoolers , 2016, CHI.

[23]  S. Gori,et al.  Action Video Games Make Dyslexic Children Read Better , 2013, Current Biology.

[24]  Cheryl K. Olson.  Children's Motivations for Video Game Play in the Context of Normal Development , 2010 .

[25]  Mohammad Obaid,et al.  "Feed the Fish": an affect-aware game , 2008, IE.

[26]  Arnold J. Wilkins,et al.  How Many Colours Are Necessary to Increase the Reading Speed of Children with Visual Stress? A Comparison of Two Systems , 2007 .

[27]  Pia Rotshtein,et al.  Identification of Emotional Facial Expressions: Effects of Expression, Intensity, and Sex on Eye Gaze , 2016, PloS one.

[28]  Min Fan,et al.  PhonoBlocks: A Tangible System for Supporting Dyslexic Children Learning to Read , 2015, TEI.

[29]  E. Vesterinen,et al.  Affective Computing , 2009, Encyclopedia of Biometrics.

[30]  A. Paivio,et al.  Dual coding theory and education , 1991 .

[31]  Virginia W. Berninger,et al.  Reading and Writing Acquisition , 1996 .

[32]  D. Dicheva,et al.  Gamifying education: what is known, what is believed and what remains uncertain: a critical review , 2017, International Journal of Educational Technology in Higher Education.

[33]  Connie Malamed,et al.  Book Review: 'The Gamification of Learning and Instruction: Game-Based Methods and Strategies For Training And Education' by Karl Kapp , 2012, ELERN.

[34]  E. Goffman.  On face-work; an analysis of ritual elements in social interaction. , 1955, Psychiatry.

[35]  Scott E. Caplan,et al.  Who plays, how much, and why? Debunking the stereotypical gamer profile , 2008, J. Comput. Mediat. Commun..

[36]  Usha Goswami,et al.  Visual attention span deficits and assessing causality in developmental dyslexia , 2015, Nature Reviews Neuroscience.

[37]  Jaynelle F Stichler.  Research, Research-Informed Design, Evidence-Based Design , 2016, HERD.

[38]  Emily Sarah Cramer,et al.  A Code of Many Colours: Rationale, Validation and Requirements for a Sound-Based Letter Colour-Code that Might Support Some Children with Dyslexia in Spelling Certain Words , 2015 .

[39]  Cynthia Breazeal,et al.  Affective Personalization of a Social Robot Tutor for Children's Second Language Skills , 2016, AAAI.

[40]  C. S. Green,et al.  Action video game modifies visual selective attention , 2003, Nature.

[41]  Clifford Nass,et al.  The media equation - how people treat computers, television, and new media like real people and places , 1996 .

[42]  S. Valdois,et al.  Developmental dyslexia: The visual attention span deficit hypothesis , 2007, Cognition.

[43]  Albert Ali Salah,et al.  Contrasting and Combining Least Squares Based Learners for Emotion Recognition in the Wild , 2015, ICMI.

[44]  J. Talcott,et al.  Adults with dyslexia exhibit large effects of crowding, increased dependence on cues, and detrimental effects of distractors in visual search tasks , 2011, Neuropsychologia.

[45]  Elly Singer,et al.  Coping with academic failure, a study of Dutch children with dyslexia. , 2008, Dyslexia.

[46]  S. Valdois,et al.  The visual attention span deficit in dyslexia is visual and not verbal , 2012, Cortex.

[47]  E. Goffman.  On face-work; an analysis of ritual elements in social interaction. , 1955, Psychiatry.

[48]  Tim Leonard,et al.  A Profile of the Labour Market Experiences of Adults with Disabilities among Canadians aged 15 years and older, 2012 , 2015 .

[49]  C.-H. Cheng,et al.  A mobile gamification learning system for improving the learning motivation and achievements , 2015, J. Comput. Assist. Learn..

[50]  Carman Neustaedter,et al.  Why Tangibility Matters: A Design Case Study of At-Risk Children Learning to Read and Spell , 2017, CHI.

[51]  A. Karimi,et al.  Master‟s thesis , 2011 .

[52]  Katherine A. Dougherty Stahl.  Response to Intervention: Is the Sky Falling?. , 2016 .

[53]  Jennifer L. Goeke,et al.  Orton-Gillingham and Orton-Gillingham—Based Reading Instruction , 2006 .

[54]  S. Dehaene.  Reading in the Brain: The New Science of How We Read , 2009 .

[55]  G. Gillon,et al.  Classroom phonological awareness instruction and literacy outcomes in the first year of school. , 2013, Language, speech, and hearing services in schools.

[56]  Sebastian Deterding,et al.  Gamification: designing for motivation , 2012, INTR.

[57]  J. Stein,et al.  Contrast sensitivity and coherent motion detection measured at photopic luminance levels in dyslexics and controls , 1995, Vision Research.

[58]  Cynthia Breazeal,et al.  Affect-Aware Student Models for Robot Tutors , 2016, AAMAS.

[59]  S. Valdois,et al.  Influence of the visual attention span on child reading performance: A cross-sectional study , 2009 .