Microsoft Kinect-Based Artificial Perception System for Control of Functional Electrical Stimulation Assisted Grasping

We present a computer vision algorithm that incorporates a heuristic model which mimics a biological control system for the estimation of control signals used in functional electrical stimulation (FES) assisted grasping. The developed processing software acquires the data from Microsoft Kinect camera and implements real-time hand tracking and object analysis. This information can be used to identify temporal synchrony and spatial synergies modalities for FES control. Therefore, the algorithm acts as artificial perception which mimics human visual perception by identifying the position and shape of the object with respect to the position of the hand in real time during the planning phase of the grasp. This artificial perception used within the heuristically developed model allows selection of the appropriate grasp and prehension. The experiments demonstrate that correct grasp modality was selected in more than 90% of tested scenarios/objects. The system is portable, and the components are low in cost and robust; hence, it can be used for the FES in clinical or even home environment. The main application of the system is envisioned for functional electrical therapy, that is, intensive exercise assisted with FES.

[1]  Juho Kannala,et al.  Joint Depth and Color Camera Calibration with Distortion Correction , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  R. Nathan,et al.  Upper limb functions regained in quadriplegia: a hybrid computerized neuromuscular stimulation system. , 1990, Archives of physical medicine and rehabilitation.

[3]  Xun Luo,et al.  Integration of Augmented Reality and Assistive Devices for Post-Stroke Hand Opening Rehabilitation , 2005, 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference.

[4]  Strahinja Došen,et al.  Cognitive vision system for control of dexterous prosthetic hands: Experimental evaluation , 2010, Journal of NeuroEngineering and Rehabilitation.

[5]  S Saxena,et al.  An EMG-controlled grasping system for tetraplegics. , 1995, Journal of rehabilitation research and development.

[6]  B Dejan Popovic,et al.  Control of arm movement: reaching synergies for neuroprosthesis with life-like control , 2002 .

[7]  Thierry Keller,et al.  Surface‐distributed low‐frequency asynchronous stimulation delays fatigue of stimulated muscles , 2013, Muscle & nerve.

[8]  Thierry Keller,et al.  A multi-pad electrode based functional electrical stimulation system for restoration of grasp , 2012, Journal of NeuroEngineering and Rehabilitation.

[9]  Berthold K. P. Horn,et al.  Determining Optical Flow , 1981, Other Conferences.

[10]  C. Von Hofsten.  Predictive reaching for moving objects by human infants. , 1980, Journal of experimental child psychology.

[11]  Blake Hannaford,et al.  Using Kinect TM and a Haptic Interface for Implementation of Real-Time Virtual Fixtures , 2011 .

[12]  Andrej M. Savić,et al.  Feasibility of a Hybrid Brain-Computer Interface for Advanced Functional Electrical Therapy , 2014, TheScientificWorldJournal.

[13]  Sander Oude Elberink,et al.  Accuracy and Resolution of Kinect Depth Data for Indoor Mapping Applications , 2012, Sensors.

[14]  W. Marsden.  I and J , 2012 .

[15]  T. Sinkjaer,et al.  Therapy of paretic arm in hemiplegic subjects augmented with a neural prosthesis: a cross-over study. , 2004, Canadian journal of physiology and pharmacology.

[16]  D. Popović,et al.  Distributed low‐frequency functional electrical stimulation delays muscle fatigue compared to conventional stimulation , 2010, Muscle & nerve.

[17]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[18]  Jarrett Webb,et al.  Beginning Kinect Programming with the Microsoft Kinect SDK , 2012, Apress.

[19]  Andrea Fossati,et al.  Consumer Depth Cameras for Computer Vision , 2013, Advances in Computer Vision and Pattern Recognition.

[20]  M. Markovic,et al.  Kinect in neurorehabilitation: Computer vision system for real time hand and object detection and distance estimation , 2012, 11th Symposium on Neural Network Applications in Electrical Engineering.

[21]  P. Peckham.  Motor prosthesis for restoration of upper extremity function , 1992 .

[22]  Strahinja Dosen,et al.  Control of Prehension for the Transradial Prosthesis: Natural-like Image Recognition System , 2009 .

[23]  Mirjana B Popović,et al.  Control of neural prostheses for grasping and reaching. , 2003, Medical engineering & physics.

[24]  Alex Mihailidis,et al.  The use of computer vision in an intelligent environment to support aging-in-place, safety, and independence in the home , 2004, IEEE Transactions on Information Technology in Biomedicine.

[25]  Gernot R. Müller-Putz,et al.  EURASIP Journal on Applied Signal Processing 2005:19, 3152–3155 c ○ 2005 Hindawi Publishing Corporation EEG-Based Asynchronous BCI Controls Functional Electrical Stimulation in a Tetraplegic Patient , 2004 .

[26]  Jane Burridge,et al.  Goal-orientated functional rehabilitation using Electrical Stimulation and Iterative Learning Control for motor recovery in the Upper Extremity Post-Stroke , 2013 .

[27]  Xin Yang,et al.  Controlling hand-assistive devices: utilizing electrooculography as a substitute for vision , 2013, IEEE Robotics & Automation Magazine.

[28]  Yoshihiro Muraoka,et al.  A home-based rehabilitation program for the hemiplegic upper extremity by power-assisted functional electrical stimulation , 2008, Disability and rehabilitation.

[29]  R. Stein,et al.  Neural prostheses : replacing motor function after disease or disability , 1992 .

[30]  Yao-Jen Chang,et al.  A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. , 2011, Research in developmental disabilities.

[31]  Dejan B Popovic,et al.  Restitution of reaching and grasping promoted by functional electrical therapy. , 2002, Artificial organs.

[32]  Jonathan W. Decker,et al.  Performance measurements for the Microsoft Kinect skeleton , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[33]  D. Popović,et al.  Multi-Pad Electrode for Effective Grasping: Design , 2013, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[34]  Mario Ciampi,et al.  Controller-free exploration of medical image data: Experiencing the Kinect , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[35]  Marko Markovic,et al.  Stereovision system for estimation of the grasp type for electrotherapy , 2011 .

[36]  Blake Hannaford,et al.  Using Kinect and a Haptic Interface for Implementation of Real-Time Virtual Fixture , 2011 .

[37]  M. Popovic,et al.  Automatic determination of the optimal shape of a surface electrode: Selective stimulation , 2009, Journal of Neuroscience Methods.

[38]  S.D. Iftime,et al.  Automatic determination of synergies by radial basis function artificial neural networks for the control of a neural prosthesis , 2005, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[39]  Strahinja Došen,et al.  Transradial prosthesis: artificial vision for control of prehension. , 2011, Artificial organs.

[40]  Dorin Comaniciu,et al.  Robust analysis of feature spaces: color image segmentation , 1997, Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[41]  Eric Rogers,et al.  Goal orientated stroke rehabilitation utilising electrical stimulation, iterative learning and Microsoft Kinect , 2013, 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR).

[42]  L. Schwirtlich,et al.  Feedback control of the forearm movement of tetraplegic patient based on Microsoft Kinect and multi-pad electrodes , 2013 .

[43]  Steve Mann,et al.  Using graphics devices in reverse: GPU-based Image Processing and Computer Vision , 2008, 2008 IEEE International Conference on Multimedia and Expo.

[44]  Tomás Pajdla,et al.  3D with Kinect , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[45]  T. Sinkjaer,et al.  Clinical evaluation of Functional Electrical Therapy in acute hemiplegic subjects. , 2003, Journal of rehabilitation research and development.

[46]  R. H. Nathan,et al.  An FNS-based system for generating upper limb function in the C4 quadriplegic , 1989, Medical and Biological Engineering and Computing.