Synthesizing Foot and Ankle Kinematic Characteristics for Lateral Collateral Ligament Injuries Detection

Deep learning has been applied in healthcare, where features of patients are read by the computer to assist with diagnosis and treatment. In sports medicine, kinematic characteristics of injuries need to be defined. Patients’ data are then acquired amplified to training deep learning models. In this study, we tracked motions of lower extremities in patients with lateral collateral ligament injuries of the ankle. Key kinematic characteristics of injuries were identified by comparing patients to normal individuals. The deep convolutional generative adversarial networks (DCGANs) was employed to synthesize a modest-sized labeled dataset to avoid the problems raised from using large-scale manual labeling data. We then fed a combination of real and synthesized data to train long short-term memory (LSTM) networks to detect patients with ligament injuries. The results showed that combined data yielded a better outcome, measured by classification accuracy and f1-score, than solely using the patient data or with a large quantity of synthesized single range of motion feature.

[1]  Moeness G. Amin,et al.  Synthesis of Micro-Doppler Signatures for Abnormal Gait using Multi-branch Discriminator with Embedded Kinematics , 2020, 2020 IEEE International Radar Conference (RADAR).

[2]  Kook Cho,et al.  Visual and Quantitative Evaluation of Amyloid Brain PET Image Synthesis with Generative Adversarial Network , 2020, Applied Sciences.

[3]  Hayit Greenspan,et al.  GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification , 2018, Neurocomputing.

[4]  J. Ryu,et al.  Biomechanical influences of gait patterns on knee joint: Kinematic & EMG analysis , 2020, PloS one.

[5]  Ukihide Tateishi,et al.  Breast Ultrasound Image Synthesis using Deep Convolutional Generative Adversarial Networks , 2019, Diagnostics.

[6]  Francesco Fioranelli,et al.  Unsupervised Learning Using Generative Adversarial Networks on micro-Doppler spectrograms , 2019, 2019 16th European Radar Conference (EuRAD).

[7]  M. H. Leonard.  Injuries of the lateral ligaments of the ankle; a clinical and experimental study. , 1949, The Journal of bone and joint surgery. American volume.

[8]  Paul John Taylor,et al.  Digital Filtering of Three-Dimensional Lower Extremity Kinematics: an Assessment , 2013, Journal of human kinetics.

[9]  Fuquan Zhang,et al.  A Novel Computer-Aided Diagnosis Scheme on Small Annotated Set: G2C-CAD , 2019, BioMed research international.

[10]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[11]  Julie Stebbins,et al.  Ground reaction forces and lower-limb joint kinetics of turning gait in typically developing children. , 2014, Journal of biomechanics.

[12]  Jingsong Xu,et al.  VT-GAN: View Transformation GAN for Gait Recognition Across Views , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[13]  Mohsen Guizani,et al.  A Decade of Internet of Things: Analysis in the Light of Healthcare Applications , 2019, IEEE Access.

[14]  G F Harris,et al.  A system for the analysis of foot and ankle kinematics during gait. , 1996, IEEE transactions on rehabilitation engineering : a publication of the IEEE Engineering in Medicine and Biology Society.

[15]  S. Wolf,et al.  Foot kinematics in walking on a level surface and on stairs in patients with hallux rigidus before and after cheilectomy , 2014, Journal of Foot and Ankle Research.

[16]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[17]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[18]  Hoon Kim,et al.  Number of Segments Within Musculoskeletal Foot Models Influences Ankle Kinematics and Strains of Ligaments and Muscles , 2019, Journal of orthopaedic research : official publication of the Orthopaedic Research Society.

[19]  M. Descarreaux,et al.  Kinematic, kinetic and electromyographic differences between young adults with and without chronic ankle instability during walking. , 2020, Journal of electromyography and kinesiology : official journal of the International Society of Electrophysiological Kinesiology.

[20]  Shiqi Yu,et al.  GaitGAN: Invariant Gait Feature Extraction Using Generative Adversarial Networks , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[21]  Hongming Shan,et al.  Multi-Task GANs for View-Specific Feature Learning in Gait Recognition , 2019, IEEE Transactions on Information Forensics and Security.

[22]  Esther Luna Colombini,et al.  Parkinson’s Disease EMG Data Augmentation and Simulation with DCGANs and Style Transfer , 2020, Sensors.

[23]  Wu Liu,et al.  Beyond View Transformation: Cycle-Consistent Global and Partial Perception Gan for View-Invariant Gait Recognition , 2018, 2018 IEEE International Conference on Multimedia and Expo (ICME).

[24]  P. Mildenberger,et al.  Structured report data can be used to develop deep learning algorithms: a proof of concept in ankle radiographs , 2019, Insights into imaging.

[25]  Yasushi Makihara,et al.  Make the Bag Disappear: Carrying Status-invariant Gait-based Human Age Estimation using Parallel Generative Adversarial Networks , 2019, 2019 IEEE 10th International Conference on Biometrics Theory, Applications and Systems (BTAS).

[26]  A. Leardini,et al.  Modifying the Rizzoli foot model to improve the diagnosis of pes-planus: application to kinematics of feet in teenagers , 2014, Journal of Foot and Ankle Research.

[27]  Zhelong Wang,et al.  Body Sensor Network-Based Gait Quality Assessment for Clinical Decision-Support via Multi-Sensor Fusion , 2019, IEEE Access.

[28]  Yue Zhu,et al.  Gait Energy Image Restoration Using Generative Adversarial Networks , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[29]  Hayit Greenspan,et al.  Cross-Modality Synthesis from CT to PET using FCN and GAN Networks for Improved Automated Lesion Detection , 2018, Eng. Appl. Artif. Intell..

[30]  Costantino Grana,et al.  Augmenting data with GANs to segment melanoma skin lesions , 2019, Multimedia Tools and Applications.

[31]  S. Wolf,et al.  Kinematic differences between normal and low arched feet in children using the Heidelberg foot measurement method. , 2010, Gait & posture.

[32]  A. Leardini,et al.  Rear-foot, mid-foot and fore-foot motion during the stance phase of gait. , 2007, Gait & posture.

[33]  S I Wolf,et al.  The Heidelberg foot measurement method: development, description and assessment. , 2006, Gait & posture.

[34]  Qingguo Li,et al.  Two Shank-Mounted IMUs-Based Gait Analysis and Classification for Neurological Disease Patients , 2020, IEEE Robotics and Automation Letters.

[35]  Junichi Yamagishi,et al.  An RGB Gait Anonymization Model for Low-Quality Silhouettes , 2019, 2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC).

[36]  Laureta Hajderanj,et al.  A New Supervised t-SNE with Dissimilarity Measure for Effective Data Visualization and Classification , 2019, ICSIE.

[37]  A. Tsubahara,et al.  Motor activation is modulated by visual experience during cyclic gait observation: A transcranial magnetic stimulation study , 2020, PloS one.

[38]  Bin Yang,et al.  Towards Adversarial Denoising of Radar Micro-Doppler Signatures , 2018, 2019 International Radar Conference (RADAR).

[39]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[40]  Alejandro F. Frangi,et al.  Retinal Image Synthesis and Semi-Supervised Learning for Glaucoma Assessment , 2019, IEEE Transactions on Medical Imaging.

[41]  Xinhua Liu,et al.  Bidirectional Face Aging Synthesis Based on Improved Deep Convolutional Generative Adversarial Networks , 2019, Inf..

[42]  S. Wolf,et al.  Tibialis posterior tendon transfer corrects the foot drop component of cavovarus foot deformity in Charcot-Marie-Tooth disease. , 2014, The Journal of bone and joint surgery. American volume.

[43]  Shahrokh Valaee,et al.  Synthesizing Chest X-Ray Pathology for Training Deep Convolutional Neural Networks , 2019, IEEE Transactions on Medical Imaging.

[44]  Gerald Harris,et al.  Quantitative characterization of gait kinematics in patients with hallux rigidus using the Milwaukee foot model , 2008, Journal of orthopaedic research : official publication of the Orthopaedic Research Society.

[45]  A. Leardini,et al.  Comparing the kinematic output of the Oxford and Rizzoli Foot Models during normal gait and voluntary pathological gait in healthy adults. , 2020, Gait & posture.

[46]  Mark Goadrich,et al.  The relationship between Precision-Recall and ROC curves , 2006, ICML.

[47]  Tomoyuki Fujioka,et al.  Virtual Interpolation Images of Tumor Development and Growth on Breast Ultrasound Image Synthesis With Deep Convolutional Generative Adversarial Networks , 2020, Journal of ultrasound in medicine : official journal of the American Institute of Ultrasound in Medicine.

[48]  Héctor F. Satizábal,et al.  DCGAN Model Used To Generate Body Gestures On a Human-Humanoid Interaction System , 2019, 2019 IEEE Colombian Conference on Applications in Computational Intelligence (ColCACI).

[49]  Wei Fang,et al.  Gesture Recognition Based on CNN and DCGAN for Calculation and Text Output , 2019, IEEE Access.

[50]  Fanfan Ma,et al.  New Gradient-Weighted Adaptive Gradient Methods With Dynamic Constraints , 2020, IEEE Access.

[51]  Jingsong Xu,et al.  VN-GAN: Identity-preserved Variation Normalizing GAN for Gait Recognition , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[52]  Esther Luna Colombini,et al.  Parkinson's Disease EMG Data Augmentation and Simulation with DCGANs and Style Transfer , 2020, Sensors.

[53]  C. Dombroski,et al.  Reliability of a multi-segment foot model in a neutral cushioning shoe during treadmill walking , 2018, Journal of Foot and Ankle Research.