Guided Embodiment and Potential Applications of Tutor Systems in Language Instruction and Rehabilitation

Intelligent tutor systems (ITSs) in mobile devices take us through learning tasks and make learning ubiquitous, autonomous, and at low cost (Nye, 2015). In this paper, we describe guided embodiment as an ITS essential feature for second language learning (L2) and aphasia rehabilitation (ARe) that enhances efficiency in the learning process. In embodiment, cognitive processes, here specifically language (re)learning are grounded in actions and gestures (Pecher and Zwaan, 2005; Fischer and Zwaan, 2008; Dijkstra and Post, 2015). In order to guide users through embodiment, ITSs must track action and gesture, and give corrective feed-back to achieve the users' goals. Therefore, sensor systems are essential to guided embodiment. In the next sections, we describe sensor systems that can be implemented in ITS for guided embodiment.

[1]  M. Macedonia,et al.  Exploring the Neural Representation of Novel Words Learned through Enactment in a Word Recognition Task , 2016, Front. Psychol..

[2]  Gerhard Tröster,et al.  Gestures are strings: efficient online gesture spotting and classification using string matching , 2007, BODYNETS.

[3]  Reinhard Oppermann,et al.  User-interface Design , 2002 .

[4]  L. Rabiner,et al.  An introduction to hidden Markov models , 1986, IEEE ASSP Magazine.

[5]  José Maria Fernandes,et al.  NeuroKinect: A Novel Low-Cost 3Dvideo-EEG System for Epileptic Seizure Motion Quantification , 2016, PloS one.

[6]  George Lakoff,et al.  Explaining Embodied Cognition Results , 2012, Top. Cogn. Sci..

[7]  Enzo Pasquale Scilingo,et al.  SMART TEXTILES FOR WEARABLE MOTION CAPTURE SYSTEMS , 2002 .

[8]  David A. Copland,et al.  Mobile computing technology and aphasia: An integrated review of accessibility and potential uses , 2013 .

[9]  John M. Carroll,et al.  Human-Computer Interaction , 2014, Lecture Notes in Computer Science.

[10]  M. Duff,et al.  Gestures make memories, but what kind? Patients with impaired procedural memory display disruptions in gesture production and comprehension , 2015, Front. Hum. Neurosci..

[11]  Osman Ipsiroglu,et al.  3D detection of periodic limb movements in sleep , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[12]  Communicative effectiveness of pantomime gesture in people with aphasia. , 2017, International journal of language & communication disorders.

[13]  V. Melissa Holland,et al.  Intelligent Language Tutors : Theory Shaping Technology , 2013 .

[14]  G. Szabo,et al.  Using mobile technology with individuals with aphasia: native iPad features and everyday apps. , 2014, Seminars in speech and language.

[15]  Rolf A. Zwaan,et al.  Grounding Cognition: Introduction to Grounding Cognition: The Role of Perception and Action in Memory, Language, and Thinking , 2005 .

[16]  HiltonAdrian,et al.  A survey of advances in vision-based human motion capture and analysis , 2006 .

[17]  Richard P. Lippmann,et al.  An introduction to computing with neural nets , 1987 .

[18]  J. Engelkamp.  Memory for actions , 1998 .

[19]  W. Buxton.  Human-Computer Interaction , 1988, Springer Berlin Heidelberg.

[20]  Jake K. Aggarwal,et al.  Human motion analysis: a review , 1997, Proceedings IEEE Nonrigid and Articulated Motion Workshop.

[21]  Sebastian Thrun,et al.  Real time motion capture using a single time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[22]  J. Fodor.  The Modularity of mind. An essay on faculty psychology , 1986 .

[23]  Daniel Roetenberg,et al.  Inertial and magnetic sensing of human motion , 2006 .

[24]  Lars-Göran Nilsson,et al.  Memory effects of motor activation in subject-performed tasks and sign language , 2003, Psychonomic bulletin & review.

[25]  Katherine E. Twomey,et al.  What's on the Inside Counts: A Grounded Account of Concept Acquisition and Development , 2016, Front. Psychol..

[26]  Wolfgang Klimesch,et al.  Long‐Term Effects of Gestures on Memory for Foreign Language Words Trained in the Classroom , 2014 .

[27]  Biing-Hwang Juang,et al.  Fundamentals of speech recognition , 1993, Prentice Hall signal processing series.

[28]  F. Tomaiuolo,et al.  Action observation as a useful approach for enhancing recovery of verb production: new evidence from aphasia. , 2013, European journal of physical and rehabilitation medicine.

[29]  Hans-Peter Seidel,et al.  Marker-less Deformable Mesh Tracking for Human Shape and Motion Capture , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[30]  C. Caltagirone,et al.  Options to enhance recovery from aphasia by means of non-invasive brain stimulation and action observation therapy , 2014, Expert review of neurotherapeutics.

[31]  K. Bergmann.  Towards gesture-based literacy training with a virtual agent , 2015 .

[32]  Jacquie Kurland,et al.  iPractice: Piloting the Effectiveness of a Tablet-Based Home Practice Program in Aphasia Treatment , 2014, Seminars in Speech and Language.

[33]  Rolf A. Zwaan,et al.  Embodied Language: A Review of the Role of the Motor System in Language Comprehension , 2008, Quarterly journal of experimental psychology.

[34]  B. Caputo,et al.  Recognizing human actions: a local SVM approach , 2004, Proceedings of the 17th International Conference on Pattern Recognition, 2004. ICPR 2004..

[35]  FriedemannPulvermüller,et al.  Constraint-Induced Therapy of Chronic Aphasia After Stroke , 2001 .

[36]  Ling Shao,et al.  Enhanced Computer Vision With Microsoft Kinect Sensor: A Review , 2013, IEEE Transactions on Cybernetics.

[37]  P. Pexman,et al.  Development of Embodied Word Meanings: Sensorimotor Effects in Children’s Lexical Processing , 2016, Front. Psychol..

[38]  Norman I. Badler,et al.  Digital Representations of Human Movement , 1979, CSUR.

[39]  M. Lavoie,et al.  Treatment of verb anomia in aphasia: efficacy of self-administered therapy using a smart tablet , 2016, Neurocase.

[40]  Manuela Macedonia,et al.  Bringing back the body into the mind: gestures enhance word learning in foreign language , 2014, Front. Psychol..

[41]  Rémi Ronfard,et al.  A survey of vision-based methods for action representation, segmentation and recognition , 2011, Comput. Vis. Image Underst..

[42]  Jake K. Aggarwal,et al.  Human Activity Recognition , 2005, PReMI.

[43]  Hubert D. Zimmer,et al.  Memory of self-performed tasks: Self-performing during recognition , 1994, Memory & cognition.

[44]  J. Fodor.  The Modularity of mind. An essay on faculty psychology , 1986 .

[45]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[46]  B. Rockstroh,et al.  Constraint-Induced Therapy of Chronic Aphasia After Stroke , 2001, Stroke.

[47]  Lars-Göran Nilsson,et al.  Implicit memory and the enactment of verbal instructions , 1989 .

[48]  W. Hinzen,et al.  The linguistic roots of natural pedagogy , 2015, Front. Psychol..

[49]  A. Borghi,et al.  Grounding Abstractness: Abstract Concepts and the Activation of the Mouth , 2016, Front. Psychol..

[50]  R. L. Cohen,et al.  On the generality of some memory laws , 1981 .

[51]  Hubert D. Zimmer,et al.  Motor programs and their relation to semantic memory. , 1985 .

[52]  S. Mitra,et al.  Gesture Recognition: A Survey , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[53]  Hubert D. Zimmer,et al.  Memory for Action: A Distinct Form of Episodic Memory? , 2001 .

[54]  Mathieu Barnachon,et al.  Ongoing human action recognition with motion capture , 2014, Pattern Recognit..

[55]  Adrian Hilton,et al.  A survey of advances in vision-based human motion capture and analysis , 2006, Comput. Vis. Image Underst..

[56]  Jake K. Aggarwal,et al.  Human activity recognition from 3D data: A review , 2014, Pattern Recognit. Lett..

[57]  Harold W. Thimbleby,et al.  User interface design , 1990, ACM Press Frontier Series.

[58]  Benjamin Nye,et al.  Intelligent Tutoring Systems by and for the Developing World: A Review of Trends and Approaches for Educational Technology in a Global Context , 2014, International Journal of Artificial Intelligence in Education.

[59]  Peter H. Veltink,et al.  Measuring orientation of human body segments using miniature gyroscopes and accelerometers , 2005, Medical and Biological Engineering and Computing.

[60]  J. Birnholz,et al.  Fetal movement patterns: a possible means of defining neurologic developmental milestones in utero. , 1978, AJR. American journal of roentgenology.

[61]  Miranda L. Rose,et al.  Releasing the constraints on aphasia therapy: the positive impact of gesture and multimodality treatments. , 2013, American journal of speech-language pathology.

[62]  B. Bahrami,et al.  Coming of age: A review of embodiment and the neuroscience of semantics , 2012, Cortex.

[63]  Mario Fernando Montenegro Campos,et al.  Online gesture recognition from pose kernel learning and decision forests , 2014, Pattern Recognit. Lett..

[64]  Lysanne S. Post,et al.  Mechanisms of embodiment , 2015, Front. Psychol..

[65]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[66]  Meinard Müller,et al.  Motion templates for automatic classification and retrieval of motion capture data , 2006, SCA '06.

[67]  Friedemann Pulvermüller,et al.  Brain connections of words, perceptions and actions: A neurobiological model of spatio-temporal semantic activation in the human cortex , 2017, Neuropsychologia.

[68]  Tara N. Sainath,et al.  Deep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups , 2012, IEEE Signal Processing Magazine.

[69]  D. Roetenberg,et al.  Xsens MVN: Full 6DOF Human Motion Tracking Using Miniature Inertial Sensors , 2009 .

[70]  Tilbe Göksun,et al.  Spontaneous gesture and spatial language: Evidence from focal brain injury , 2015, Brain and Language.

[71]  Ronald Poppe,et al.  A survey on vision-based human action recognition , 2010, Image Vis. Comput..

[72]  Ursula Dresdner,et al.  Computation Finite And Infinite Machines , 2016 .

[73]  Joachim Hornegger,et al.  Gesture recognition with a Time-Of-Flight camera , 2008, Int. J. Intell. Syst. Technol. Appl..

[74]  Friedemann Pulvermüller,et al.  The Neuroscience of Language: On Brain Circuits of Words and Serial Order , 2003 .

[75]  Andrei M. Shkel.  Precision navigation and timing enabled by microtechnology: Are we there yet? , 2010, 2010 IEEE Sensors.

[76]  V. M. Lubecke,et al.  10 GHz Doppler radar sensing of respiration and heart movement , 2002, Proceedings of the IEEE 28th Annual Northeast Bioengineering Conference (IEEE Cat. No.02CH37342).

[77]  G. Morgan,et al.  Language and iconic gesture use in procedural discourse by speakers with aphasia , 2015, Aphasiology.

[78]  Jake K. Aggarwal,et al.  Tracking Human Motion in Structured Environments Using a Distributed-Camera System , 1999, IEEE Trans. Pattern Anal. Mach. Intell..

[79]  E. Maris,et al.  Two Sides of the Same Coin , 2010, Psychological science.

[80]  Hellmuth Obrig,et al.  Lexical learning in mild aphasia: Gesture benefit depends on patholinguistic profile and lesion pattern , 2013, Cortex.

[81]  Fei Han,et al.  Space-Time Representation of People Based on 3D Skeletal Data: A Review , 2016, Comput. Vis. Image Underst..

[82]  Angelo M. Sabatini,et al.  Estimating Three-Dimensional Orientation of Human Body Parts by Inertial/Magnetic Sensing , 2011, Sensors.

[83]  D. McDermott.  LANGUAGE OF THOUGHT , 2012 .