Enabling Detailed, Biophysics-Based Skeletal Muscle Models on HPC Systems

Realistic simulations of detailed, biophysics-based, multi-scale models often require very high resolution and, thus, large-scale compute facilities. Existing simulation environments, especially for biomedical applications, are typically designed to allow for high flexibility and generality in model development. Flexibility and model development, however, are often a limiting factor for large-scale simulations. Therefore, new models are typically tested and run on small-scale compute facilities. By using a detailed biophysics-based, chemo-electromechanical skeletal muscle model and the international open-source software library OpenCMISS as an example, we present an approach to upgrade an existing muscle simulation framework from a moderately parallel version toward a massively parallel one that scales both in terms of problem size and in terms of the number of parallel processes. For this purpose, we investigate different modeling, algorithmic and implementational aspects. We present improvements addressing both numerical and parallel scalability. In addition, our approach includes a novel visualization environment which is based on the MegaMol framework and is capable of handling large amounts of simulated data. We present the results of a number of scaling studies at the Tier-1 supercomputer HazelHen at the High Performance Computing Center Stuttgart (HLRS). We improve the overall runtime by a factor of up to 2.6 and achieve good scalability on up to 768 cores.

[1]  R. Šimkus,et al.  Computer simulation of the steady state currents at enzyme doped carbon paste electrodes , 2005 .

[2]  D Hawkins,et al.  A comprehensive approach for studying muscle-tendon mechanics. , 1994, Journal of biomechanical engineering.

[3]  William Gropp,et al.  Efficient Management of Parallelism in Object-Oriented Numerical Software Libraries , 1997, SciTools.

[4]  C. C. Law,et al.  ParaView: An End-User Tool for Large-Data Visualization , 2005, The Visualization Handbook.

[5]  B Calvo,et al.  A 3D electro-mechanical continuum model for simulating skeletal muscle contraction. , 2013, Journal of theoretical biology.

[6]  Simone Scacchi,et al.  Parallel multilevel solvers for the cardiac electro-mechanical coupling , 2015 .

[7]  Fan Zhang,et al.  In Situ Processing , 2012, High Performance Visualization.

[8]  Patrick Amestoy,et al.  A Fully Asynchronous Multifrontal Solver Using Distributed Dynamic Scheduling , 2001, SIAM J. Matrix Anal. Appl..

[9]  Thomas Ertl,et al.  Challenges and Opportunities using Software-defined Visualization in MegaMol , 2017 .

[10]  W Herzog,et al.  Length dependence of active force production in skeletal muscle. , 1999, Journal of applied physiology.

[11]  Peter J. Hunter,et al.  FieldML: concepts and implementation , 2009, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[12]  Oliver Röhrle,et al.  Modeling the Chemoelectromechanical Behavior of Skeletal Muscle Using the Parallel Open-Source Software Library OpenCMISS , 2013, Comput. Math. Methods Medicine.

[13]  K B Campbell,et al.  Stiffness-distortion sarcomere model for muscle simulation. , 1999, Journal of applied physiology.

[14]  D. Winter,et al.  Models of recruitment and rate coding organization in motor-unit pools. , 1993, Journal of neurophysiology.

[15]  Hank Childs,et al.  VisIt: An End-User Tool for Visualizing and Analyzing Very Large Data , 2011 .

[16]  Dario Farina,et al.  An analytical model for surface EMG generation in volume conductors with smooth conductivity variations , 2006, IEEE Transactions on Biomedical Engineering.

[17]  Oliver Röhrle,et al.  A multi-scale continuum model of skeletal muscle mechanics predicting force enhancement based on actin–titin interaction , 2016, Biomechanics and modeling in mechanobiology.

[18]  Todd A. Kuiken,et al.  A multiple-layer finite-element model of the surface EMG signal , 2002, IEEE Transactions on Biomedical Engineering.

[19]  Oliver Röhrle,et al.  A multiscale chemo-electro-mechanical skeletal muscle model to analyze muscle contraction and force generation for different muscle fiber arrangements , 2014, Front. Physiol..

[20]  Aaron Knoll,et al.  OSPRay - A CPU Ray Tracing Framework for Scientific Visualization , 2017, IEEE Transactions on Visualization and Computer Graphics.

[21]  Kwai L. Wong,et al.  A Fully Coupled Model for Electromechanics of the Heart , 2012, Comput. Math. Methods Medicine.

[22]  Hiroshi Nishiura,et al.  Age-Dependent Estimates of the Epidemiological Impact of Pandemic Influenza (H1N1-2009) in Japan , 2013, Comput. Math. Methods Medicine.

[23]  Oliver Röhrle,et al.  Predicting electromyographic signals under realistic conditions using a multiscale chemo–electro–mechanical finite element model , 2015, Interface Focus.

[24]  M. Hestenes,et al.  Methods of conjugate gradients for solving linear systems , 1952 .

[25]  Stefan Schamberger,et al.  Partitioning finite element meshes using space-filling curves , 2005, Future Gener. Comput. Syst..

[26]  Patrick Amestoy,et al.  Hybrid scheduling for the parallel solution of linear systems , 2006, Parallel Comput..

[27]  O Röhrle,et al.  A two-muscle, continuum-mechanical forward simulation of the upper limb , 2017, Biomechanics and modeling in mechanobiology.

[28]  Catherine M Lloyd,et al.  CellML: its future, present and past. , 2004, Progress in biophysics and molecular biology.

[29]  Markus Böl,et al.  Micromechanical modelling of skeletal muscles based on the finite element method , 2008, Computer Methods in Biomechanics and Biomedical Engineering.

[30]  Heye Zhang,et al.  OpenCMISS: a multi-physics & multi-scale computational infrastructure for the VPH/Physiome project. , 2011, Progress in biophysics and molecular biology.

[31]  Thomas Klotz,et al.  A continuum-mechanical skeletal muscle model including actin-titin interaction predicts stable contractions on the descending limb of the force-length relation , 2017, PLoS Comput. Biol..

[32]  N. Dimitrova,et al.  Precise and fast calculation of the motor unit potentials detected by a point and rectangular plate electrode. , 1998, Medical engineering & physics.

[33]  J. B. Davidson,et al.  A Physiologically Based, Multi-Scale Model of Skeletal Muscle Structure and Function , 2012, Front. Physio..

[34]  Gary L. Miller,et al.  Automatic Mesh Partitioning , 1992 .

[35]  Roberto Merletti,et al.  Biophysics of the Generation of EMG Signals , 2004 .

[36]  D. Farina,et al.  Decorrelation of cortical inputs and motoneuron output. , 2011, Journal of neurophysiology.

[37]  O Röhrle,et al.  Gradient-based optimization with B-splines on sparse grids for solving forward-dynamics simulations of three-dimensional, continuum-mechanical musculoskeletal system models. , 2018, International journal for numerical methods in biomedical engineering.

[38]  Y. Saad,et al.  GMRES: a generalized minimal residual algorithm for solving nonsymmetric linear systems , 1986 .

[39]  Roberto Merletti,et al.  Electromyography. Physiology, engineering and non invasive applications , 2005 .

[40]  M D Binder,et al.  Computer simulation of the steady-state input-output function of the cat medial gastrocnemius motoneuron pool. , 1991, Journal of neurophysiology.

[41]  Guillaume Houzeaux,et al.  Coupled electromechanical model of the heart: Parallel finite element formulation , 2012, International journal for numerical methods in biomedical engineering.

[42]  Dario Farina,et al.  An integrated model of the neuromuscular system , 2013, 2013 6th International IEEE/EMBS Conference on Neural Engineering (NER).

[43]  Luca Mesin,et al.  Volume conductor models in surface electromyography: Computational techniques , 2013, Comput. Biol. Medicine.

[44]  B. Feinstein,et al.  Morphologic studies of motor units in normal human muscles. , 1955, Acta anatomica.

[45]  Thomas Ertl,et al.  Hardware-Accelerated Glyphs for Mono- and Dipoles in Molecular Dynamics Visualization , 2005, EuroVis.

[46]  Rogério Rodrigues Lima Cisi,et al.  Simulation system of spinal cord motor nuclei and associated nerves and muscles, in a Web-based architecture , 2008, Journal of Computational Neuroscience.

[47]  Gennadiy Nikishkov,et al.  Finite Element Model , 2010 .

[48]  S. Delp,et al.  A 3D model of muscle reveals the causes of nonuniform strains in the biceps brachii. , 2005, Journal of biomechanics.

[49]  D. Leroith,et al.  Skeletal muscle. , 2005, Advances in experimental medicine and biology.

[50]  Onkar Sahni,et al.  Controlling Unstructured Mesh Partitions for Massively Parallel Simulations , 2010, SIAM J. Sci. Comput..

[51]  Andrew J. Pullan,et al.  Mathematically Modelling the Electrical Activity of the Heart: From Cell to Body Surface and Back Again , 2005 .

[52]  M. Karhanek,et al.  An allosteric model of the molecular interactions of excitation- contraction coupling in skeletal muscle , 1993, The Journal of general physiology.

[53]  Valerio Pascucci,et al.  CPU ray tracing large particle data with balanced P-k-d trees , 2015, 2015 IEEE Scientific Visualization Conference (SciVis).

[54]  John B. Davidson,et al.  A mathematical model of fatigue in skeletal muscle force contraction , 2007, Journal of Muscle Research and Cell Motility.

[55]  John B. Davidson,et al.  Bridging Scales: A Three-Dimensional Electromechanical Finite Element Model of Skeletal Muscle , 2008, SIAM J. Sci. Comput..

[56]  Thomas Ertl,et al.  MegaMol—A Prototyping Framework for Particle-Based Visualization , 2015, IEEE Transactions on Visualization and Computer Graphics.

[57]  Oliver Röhrle,et al.  POD-DEIM reduction of computational EMG models , 2017, J. Comput. Sci..

[58]  R. Blickhan,et al.  A finite-element model for the mechanical analysis of skeletal muscles. , 2000, Journal of theoretical biology.

[59]  A. Pullan,et al.  Three-dimensional finite element modelling of muscle forces during mastication. , 2007, Journal of biomechanics.

[60]  A. Tveito,et al.  An operator splitting method for solving the bidomain equations coupled to a volume conductor model for the torso. , 2005, Mathematical biosciences.

[61]  Dario Farina,et al.  A novel approach for precise simulation of the EMG signal detected by surface electrodes , 2001, IEEE Trans. Biomed. Eng..

[62]  F. Zajac Muscle and tendon: properties, models, scaling, and application to biomechanics and motor control. , 1989, Critical reviews in biomedical engineering.

[63]  Viatcheslav Gurev,et al.  A high-resolution computational model of the deforming human heart , 2015, Biomechanics and modeling in mechanobiology.

[64]  A. Huxley,et al.  The variation in isometric tension with sarcomere length in vertebrate muscle fibres , 1966, The Journal of physiology.

[65]  A. Garfinkel,et al.  An advanced algorithm for solving partial differential equation in cardiac conduction , 1999, IEEE Transactions on Biomedical Engineering.

[66]  A. Hodgkin,et al.  A quantitative description of membrane current and its application to conduction and excitation in nerve , 1952, The Journal of physiology.