Mobile music touch: using haptic stimulation for passive rehabilitation and learning

Hand rehabilitation after injury or illness may allow a patient to regain full or at least partial use of a limb. However, rehabilitation often requires the patient to perform multiple repetitions of motions. While absolutely essential to regaining usage, such exercises are not always mentally engaging or enjoyable for the patient. The loss or degradation of the use of the hands can cause considerable loss of independence. In this dissertation, we present Mobile Music Touch (MMT), a wireless glove paired with a computing device, such as a laptop, smart phone, or MP3 player. The MMT system plays a song, while also “tapping” the fingers using vibration motors to indicate the correct finger to use to play the song on a piano keyboard. Learning a new skill or activity without active focus, an idea called Passive Haptic Learning (PHL) may allow an individual to learn one skill through their sense of touch while performing another, unrelated activity. Most rehabilitation activities are active in nature, requiring the focused participation of the injured person. Passive rehabilitation is the idea that some technologies and activities may bring about beneficial changes without the active engagement of the injured person. In order to study the concepts of PHL and PHR, we propose the Mobile Music Touch (MMT) system. We show that using passive rehabilitation in conjunction with the active rehabilitation of piano playing will bring about a greater degree of improvement in the hands than that achieved using only active rehabilitation. This dissertation research makes three unique contributions. First, we demonstrate that Passive Haptic Learning (PHL) using just the sense of touch is feasible and provides a form of learning and reinforcement of learned skills and tasks. Second, we identify the attributes and design features of a glove suited for long term wear by persons who use a manual wheelchair for mobility. Third, we show that Passive Haptic Rehabilitation (PHR) is possible using vibrotactile stimulation of the hands in persons classified as tetraplegic due to incomplete spinal cord injury.

[1]  F A GELDARD,et al.  Some neglected possibilities of communication. , 1960, Science.

[2]  James Charles Bliss,et al.  Communication via the kinesthetic and tactile senses , 1961 .

[3]  J. R. Barry PATIENT MOTIVATION FOR REHABILITATION. , 1965, The Cleft palate journal.

[4]  I. Howard,et al.  Human Spatial Orientation , 1966 .

[5]  Richard C. Atkinson,et al.  Human Memory: A Proposed System and its Control Processes , 1968, Psychology of Learning and Motivation.

[6]  Jacob Cohen Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[7]  H. E. Krugman,et al.  PASSIVE LEARNING FROM TELEVISION , 1970 .

[8]  R. C. Oldfield The assessment and analysis of handedness: the Edinburgh inventory. , 1971, Neuropsychologia.

[9]  F. Craik,et al.  Levels of Pro-cessing: A Framework for Memory Research , 1975 .

[10]  A. Freehafer,et al.  Tendon transfers to improve grasp after injuries of the cervical spinal cord. , 1974, The Journal of bone and joint surgery. American volume.

[11]  C. Clemente,et al.  Anatomy;: A regional atlas of the human body , 1975 .

[12]  [Interaction of transcranial magnetic stimulation and mechanical stimuli]. , 1988, EEG-EMG Zeitschrift fur Elektroenzephalographie, Elektromyographie und verwandte Gebiete.

[13]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[14]  P. Lachenbruch Statistical Power Analysis for the Behavioral Sciences (2nd ed.) , 1989 .

[15]  M. Sur,et al.  Cross-modal plasticity in cortical development: differentiation and specification of sensory neocortex , 1990, Trends in Neurosciences.

[16]  E. Moberg,et al.  Two-point discrimination test. A valuable part of hand surgical rehabilitation, e.g. in tetraplegia. , 1989, Scandinavian journal of rehabilitation medicine.

[17]  R. Waters,et al.  Motor and sensory recovery following complete tetraplegia. , 1993, Archives of physical medicine and rehabilitation.

[18]  R. Waters,et al.  Motor and sensory recovery following incomplete paraplegia. , 1994, Archives of physical medicine and rehabilitation.

[19]  Dimitrijević Mm,et al.  Mesh-glove. 1. A method for whole-hand electrical stimulation in upper motor neuron dysfunction. , 1994 .

[20]  M. Curtin,et al.  Development of a tetraplegic hand assessment and splinting protocol , 1994, Paraplegia.

[21]  M. Dimitrijevic,et al.  Mesh-glove. 1. A method for whole-hand electrical stimulation in upper motor neuron dysfunction. , 1994, Scandinavian journal of rehabilitation medicine.

[22]  K. Hammell Spinal Cord Injury Rehabilitation , 2014, Therapy in Practice Series.

[23]  M. Dimitrijevic,et al.  Modification of motor control of wrist extension by mesh-glove electrical afferent stimulation in stroke patients. , 1996, Archives of physical medicine and rehabilitation.

[24]  B. Lee,et al.  Use of electronic music as an occupational therapy modality in spinal cord injury rehabilitation: an occupational performance model. , 1996, The American journal of occupational therapy : official publication of the American Occupational Therapy Association.

[25]  R J Ilmoniemi,et al.  Electrophysiological evidence for cross-modal plasticity in humans with early- and late-onset blindness. , 1997, Psychophysiology.

[26]  J. M. Ollinger,et al.  Positron Emission Tomography , 2018, Handbook of Small Animal Imaging.

[27]  J A Shea,et al.  The Capabilities of Upper Extremity instrument: reliability and validity of a measure of functional limitation in tetraplegia. , 1998, Archives of physical medicine and rehabilitation.

[28]  A Curt,et al.  How does the human brain deal with a spinal cord injury? , 1998, The European journal of neuroscience.

[29]  C Kremser,et al.  Functional magnetic resonance imaging of the human motor cortex before and after whole-hand afferent electrical stimulation. , 1999, Scandinavian journal of rehabilitation medicine.

[30]  Michael Recce,et al.  A virtual reality-based exercise program for stroke rehabilitation , 2000, Assets '00.

[31]  J. Gladman Improving long-term rehabilitation. , 2000, British medical bulletin.

[32]  C. Wolfe,et al.  Qualitative analysis of stroke patients' motivation for rehabilitation , 2000, BMJ : British Medical Journal.

[33]  Samuel C. Shiflett,et al.  Effect of music therapy on mood and social interaction among individuals with acute traumatic brain injury and stroke. , 2000 .

[34]  Jennifer M. Glass,et al.  Virtually Perfect Time Sharing in Dual-Task Performance: Uncorking the Central Cognitive Bottleneck , 2001, Psychological science.

[35]  J Crosbie,et al.  Hand function of C6 and C7 tetraplegics 1–16 years following injury , 2001, Spinal Cord.

[36]  D. Meyer,et al.  Executive control of cognitive processes in task switching. , 2001, Journal of experimental psychology. Human perception and performance.

[37]  S. Shimojo,et al.  Sensory modalities are not separate modalities: plasticity and interactions , 2001, Current Opinion in Neurobiology.

[38]  J. H. van der Lee,et al.  The intra- and interrater reliability of the action research arm test: a practical test of upper extremity function in patients with stroke. , 2001, Archives of physical medicine and rehabilitation.

[39]  J Esnouf,et al.  The functional impact of the Freehand System on tetraplegic hand function. Clinical Results , 2002, Spinal Cord.

[40]  John J. Foxe,et al.  Multisensory auditory-visual interactions during early sensory processing in humans: a high-density electrical mapping study. , 2002, Brain research. Cognitive brain research.

[41]  Physical Examination of the Spine , 2003 .

[42]  W. T. Thach,et al.  How do strength, sensation, spasticity and joint individuation relate to the reaching deficits of people with chronic hemiparesis? , 2004, Brain : a journal of neurology.

[43]  Recent Trends in Spinal Cord Injury Rehabilitation Practices and Outcomes , 2004 .

[44]  Maarten J. IJzerman,et al.  Survey of the needs of patients with spinal cord injury: impact and priority for improvement in hand function in tetraplegics , 2004, Spinal Cord.

[45]  Janez Zidar,et al.  REMARKS ON NEUROCONTROL OF THE HAND AND SIGNIFICANCE OF AFFERENT INPUT , 2004 .

[46]  P. Matthews,et al.  Functional magnetic resonance imaging. , 2004, Journal of neurology, neurosurgery, and psychiatry.

[47]  K. Anderson Targeting recovery: priorities of the spinal cord-injured population. , 2004, Journal of neurotrauma.

[48]  Yoshinori Kuno,et al.  Multimodal presentation method for a dance training system , 2005, CHI Extended Abstracts.

[49]  Alex Pentland,et al.  Tactual displays for wearable computing , 2005, Personal Technologies.

[50]  Ronald A. Rensink,et al.  Change blindness: past, present, and future , 2005, Trends in Cognitive Sciences.

[51]  Joanne M Wagner,et al.  Measurement of upper-extremity function early after stroke: properties of the action research arm test. , 2006, Archives of physical medicine and rehabilitation.

[52]  D. Basso,et al.  Development of the Hand Active Sensation Test: reliability and validity. , 2006, Archives of physical medicine and rehabilitation.

[53]  P. Dario,et al.  Design strategies to improve patient motivation during robot-aided rehabilitation , 2007, Journal of NeuroEngineering and Rehabilitation.

[54]  C. Spence,et al.  The failure to detect tactile change: A tactile analogue of visual change blindness , 2006, Psychonomic bulletin & review.

[55]  S. Sahrmann,et al.  Relationships between Sensorimotor Impairments and Reaching Deficits in Acute Hemiparesis , 2006, Neurorehabilitation and neural repair.

[56]  M J Mulcahey,et al.  Assessment of upper limb in tetraplegia: considerations in evaluation and outcomes research. , 2007, Journal of rehabilitation research and development.

[57]  John Kenneth Salisbury,et al.  Haptic Feedback Enhances Force Skill Learning , 2007, Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC'07).

[58]  N. Doidge The Brain That Changes Itself , 2007 .

[59]  E. Field-Fote,et al.  Cortical Reorganization Following Bimanual Training and Somatosensory Stimulation in Cervical Spinal Cord Injury: A Case Report , 2007, Physical Therapy.

[60]  Ellen Yi-Luen Do,et al.  PianoTouch: A wearable haptic piano instruction system for passive learning of piano skills , 2008, 2008 12th IEEE International Symposium on Wearable Computers.

[61]  E. Field-Fote,et al.  Sensory stimulation augments the effects of massed practice training in persons with tetraplegia. , 2008, Archives of physical medicine and rehabilitation.

[62]  Michelle McDonnell,et al.  Action research arm test. , 2008, The Australian journal of physiotherapy.

[63]  Mark R. Cutkosky,et al.  Improving finger force control with vibrational haptic feedback for Multiple Sclerosis , 2008 .

[64]  N. Yozbatiran,et al.  A Standardized Approach to Performing the Action Research Arm Test , 2008, Neurorehabilitation and neural repair.

[65]  P. Cordo,et al.  Assisted Movement With Enhanced Sensation (AMES): Coupling Motor and Sensory to Remediate Motor Deficits in Chronic Stroke Patients , 2009, Neurorehabilitation and neural repair.

[66]  Janet van der Linden,et al.  Towards a real-time system for teaching novices correct violin bowing technique , 2009, 2009 IEEE International Workshop on Haptic Audio visual Environments and Games.

[67]  Frank Drews,et al.  Text Messaging During Simulated Driving , 2009, Hum. Factors.

[68]  Dieter Drobny,et al.  Saltate!: a sensor-based system to support dance beginners , 2009, CHI Extended Abstracts.

[69]  Erwin Schoonderwaldt,et al.  Good vibrations: Guiding body movements with vibrotactile feedback , 2009 .

[70]  Matthew Eicholtz,et al.  Mobile Music Touch: Vibration stimulus in hand rehabilitation , 2010, 2010 4th International Conference on Pervasive Computing Technologies for Healthcare.

[71]  Ellen Yi-Luen Do,et al.  Mobile music touch: mobile tactile stimulation for passive learning , 2010, CHI.

[72]  Dieter Drobny,et al.  Learning basic dance choreographies with different augmented feedback modalities , 2010, CHI Extended Abstracts.

[73]  Jun Rekimoto,et al.  PossessedHand: a hand gesture manipulation system using electrical stimuli , 2010, AH.

[74]  Thad Starner,et al.  Mobile Music Touch: The effect of primary tasks on passively learning piano sequences , 2010, International Symposium on Wearable Computers (ISWC) 2010.

[75]  Thad Starner,et al.  Buzzwear: supporting multitasking with wearable tactile displays on the wrist , 2010 .

[76]  Thad Starner,et al.  BuzzWear: alert perception in wearable tactile displays on the wrist , 2010, CHI.

[77]  Yvonne Rogers,et al.  MusicJacket: the efficacy of real-time vibrotactile feedback for learning to play the violin , 2010, CHI EA '10.

[78]  Thad Starner,et al.  Touchfire: Towards a glove-mounted tactile display for rendering temperature readings for firefighters , 2010, International Symposium on Wearable Computers (ISWC) 2010.

[79]  Yvonne Rogers,et al.  Buzzing to play: lessons learned from an in the wild study of real-time vibrotactile feedback , 2011, CHI.

[80]  Erwin Schoonderwaldt,et al.  MusicJacket—Combining Motion Capture and Vibrotactile Feedback to Teach Violin Bowing , 2011, IEEE Transactions on Instrumentation and Measurement.

[81]  Jun Rekimoto,et al.  PossessedHand: Controlling Hand Movements with Computer Output , 2011, Computer.

[82]  Jun Ueda,et al.  Wearable sensorimotor enhancer for a fingertip based on stochastic resonance , 2011, 2011 IEEE International Conference on Robotics and Automation.

[83]  Jun Rekimoto,et al.  PossessedHand: techniques for controlling human hands using electrical muscles stimuli , 2011, CHI.

[84]  Karolyn O. Babalola Brain-computer interfaces for inducing brain plasticity and motor learning: implications for brain-injury rehabilitation , 2011 .

[85]  Geoff V. Merrett,et al.  Design and qualitative evaluation of tactile devices for stroke rehabilitation , 2011 .

[86]  S. Sulaiman Cutaneous innervation of the hand , 2014 .