Entropy measures, entropy estimators, and their performance in quantifying complex dynamics: Effects of artifacts, nonstationarity, and long-range correlations.

Entropy measures are widely applied to quantify the complexity of dynamical systems in diverse fields. However, the practical application of entropy methods is challenging, due to the variety of entropy measures and estimators and the complexity of real-world time series, including nonstationarities and long-range correlations (LRC). We conduct a systematic study on the performance, bias, and limitations of three basic measures (entropy, conditional entropy, information storage) and three traditionally used estimators (linear, kernel, nearest neighbor). We investigate the dependence of entropy measures on estimator- and process-specific parameters, and we show the effects of three types of nonstationarities due to artifacts (trends, spikes, local variance change) in simulations of stochastic autoregressive processes. We also analyze the impact of LRC on the theoretical and estimated values of entropy measures. Finally, we apply entropy methods on heart rate variability data from subjects in different physiological states and clinical conditions. We find that entropy measures can only differentiate changes of specific types in cardiac dynamics and that appropriate preprocessing is vital for correct estimation and interpretation. Demonstrating the limitations of entropy methods and shedding light on how to mitigate bias and provide correct interpretations of results, this work can serve as a comprehensive reference for the application of entropy methods and the evaluation of existing studies.

[1]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[2]  C. Granger,et al.  AN INTRODUCTION TO LONG‐MEMORY TIME SERIES MODELS AND FRACTIONAL DIFFERENCING , 1980 .

[3]  R. Cohen,et al.  Power spectrum analysis of heart rate fluctuation: a quantitative probe of beat-to-beat cardiovascular control. , 1981, Science.

[4]  T. Musha,et al.  1/f Fluctuation of Heartbeat Period , 1982, IEEE Transactions on Biomedical Engineering.

[5]  P. Grassberger,et al.  Dimensions and entropies of strange attractors from a fluctuating dynamics approach , 1984 .

[6]  D. Ruelle,et al.  Ergodic theory of chaos and strange attractors , 1985 .

[7]  Farmer,et al.  Predicting chaotic time series. , 1987, Physical review letters.

[8]  C. Tsallis.  Possible generalization of Boltzmann-Gibbs statistics , 1988 .

[9]  Martin Casdagli,et al.  Nonlinear prediction of chaotic time series , 1989 .

[10]  K. Briggs.  An improved method for estimating Liapunov exponents of chaotic time series , 1990 .

[11]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[12]  A. Malliani,et al.  Cardiovascular Neural Regulation Explored in the Frequency Domain , 1991, Circulation.

[13]  Y. Ogata,et al.  Some Statistical Features of the Long-Term Variation of the Global and Regional Seismic Activity , 1991 .

[14]  A L Goldberger,et al.  Physiological time-series analysis: what does regularity quantify? , 1994, The American journal of physiology.

[15]  H. Stanley,et al.  Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. , 1995, Chaos.

[16]  J. Kurths,et al.  Quantitative analysis of heart rate variability. , 1995, Chaos.

[17]  Shlomo Havlin,et al.  Scaling behaviour of heartbeat intervals obtained by wavelet-based time-series analysis , 1996, Nature.

[18]  Anishchenko,et al.  Dynamical Entropies Applied to Stochastic Resonance , 1996, Physical review letters.

[19]  Richard T. Baillie,et al.  Long memory processes and fractional integration in econometrics , 1996 .

[20]  W. Ebeling.  Entropy, information and predictability of evolutionary systems , 1997 .

[21]  K. Martinás.  Entropy and information , 1997 .

[22]  P. Ponikowski,et al.  Depressed heart rate variability as an independent predictor of death in chronic congestive heart failure secondary to ischemic or idiopathic dilated cardiomyopathy. , 1997, The American journal of cardiology.

[23]  G. C. Butler,et al.  Fractal component of variability of heart rate and systolic blood pressure in congestive heart failure. , 1997, Clinical science.

[24]  H. Nagaraja,et al.  Heart rate variability: origins, methods, and interpretive caveats. , 1997, Psychophysiology.

[25]  S. Havlin,et al.  Indication of a Universal Persistence Law Governing Atmospheric Variability , 1998 .

[26]  Giuseppe Baselli,et al.  Measuring regularity by means of a corrected conditional entropy in sympathetic outflow , 1998, Biological Cybernetics.

[27]  Walter Willinger,et al.  Stock market prices and long-range dependence , 1999, Finance Stochastics.

[28]  L. Amaral,et al.  Multifractality in human heartbeat dynamics , 1998, Nature.

[29]  Ivanov PCh,et al.  Sleep-wake differences in scaling behavior of the human heartbeat: analysis of terrestrial and long-term space flight data. , 1999, Europhysics letters.

[30]  H. Huikuri,et al.  Altered complexity and correlation properties of R-R interval dynamics before the spontaneous onset of paroxysmal atrial fibrillation. , 1999, Circulation.

[31]  S. Havlin,et al.  Correlated and uncorrelated regions in heart-rate fluctuations during sleep. , 2000, Physical review letters.

[32]  A. Porta,et al.  Linear and non-linear 24 h heart rate variability in chronic heart failure , 2000, Autonomic Neuroscience.

[33]  J. Richman,et al.  Physiological time-series analysis using approximate entropy and sample entropy. , 2000, American journal of physiology. Heart and circulatory physiology.

[34]  Schreiber,et al.  Measuring information transfer , 2000, Physical review letters.

[35]  H. Stanley,et al.  Scale invariance in the nonstationarity of human heart rate. , 2000, Physical review letters.

[36]  H. Stanley,et al.  Effect of trends on detrended fluctuation analysis. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[37]  Andrew G. Glen,et al.  APPL , 2001 .

[38]  H. Stanley,et al.  Magnitude and sign correlations in heartbeat fluctuations. , 2000, Physical review letters.

[39]  Luís A. Nunes Amaral,et al.  From 1/f noise to multifractal cascades in heartbeat dynamics. , 2001, Chaos.

[40]  B. Pompe,et al.  Permutation entropy: a natural complexity measure for time series. , 2002, Physical review letters.

[41]  M. Saling.  The executive brain: frontal lobes and the civilized mind , 2002 .

[42]  Jeffrey M. Hausdorff,et al.  Fractal dynamics in physiology: Alterations with disease and aging , 2002, Proceedings of the National Academy of Sciences of the United States of America.

[43]  P. Lavie,et al.  Correlation differences in heartbeat fluctuations during rest and exercise. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[44]  J. Victor.  Binless strategies for estimation of information from neural data. , 2002, Physical review. E, Statistical, nonlinear, and soft matter physics.

[45]  Madalena Costa,et al.  Multiscale entropy analysis of complex physiologic time series. , 2002, Physical review letters.

[46]  H. Stanley,et al.  Characterization of sleep stages by correlations in the magnitude and sign of heartbeat increments. , 2000, Physical review. E, Statistical, nonlinear, and soft matter physics.

[47]  J. Taylor,et al.  Short‐term cardiovascular oscillations in man: measuring and modelling the physiologies , 2002, The Journal of physiology.

[48]  Harvard Medical School,et al.  Effect of nonstationarities on detrended fluctuation analysis. , 2001, Physical review. E, Statistical, nonlinear, and soft matter physics.

[49]  T. Schreiber,et al.  Information transfer in continuous processes , 2002 .

[50]  R. Acharya U,et al.  Comprehensive analysis of cardiac health using heart rate signals , 2004, Physiological measurement.

[51]  P. Varotsos,et al.  Entropy in the natural time domain. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[52]  A. Kraskov,et al.  Estimating mutual information. , 2003, Physical review. E, Statistical, nonlinear, and soft matter physics.

[53]  P. Ivanov,et al.  Common scaling patterns in intertrade times of U. S. stocks. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[54]  H. Stanley,et al.  Common scale-invariant patterns of sleep-wake transitions across mammalian species. , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[55]  Madalena Costa,et al.  Multiscale entropy analysis of biological signals. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[56]  Ericka Stricklin-Parker,et al.  Ann , 2005 .

[57]  R. Radner.  PROCEEDINGS of the FOURTH BERKELEY SYMPOSIUM ON MATHEMATICAL STATISTICS AND PROBABILITY , 2005 .

[58]  H. Stanley,et al.  Effect of nonlinear filters on detrended fluctuation analysis. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[59]  P. Stein,et al.  Heart Rate Variability: Measurement and Clinical Utility , 2005, Annals of noninvasive electrocardiology : the official journal of the International Society for Holter and Noninvasive Electrocardiology, Inc.

[60]  P. Varotsos,et al.  Some properties of the entropy in the natural time. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[61]  Roberto Hornero,et al.  Interpretation of approximate entropy: analysis of intracranial pressure approximate entropy during acute intracranial hypertension , 2005, IEEE Transactions on Biomedical Engineering.

[62]  H. Stanley,et al.  Quantifying signals with power-law correlations: a comparative study of detrended fluctuation analysis and detrended moving average techniques. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[63]  P. Varotsos,et al.  Entropy of seismic electric signals: analysis in natural time under time reversal. , 2005, Physical review. E, Statistical, nonlinear, and soft matter physics.

[64]  N. Stergiou,et al.  A Novel Approach to Measure Variability in the Anterior Cruciate Ligament Deficient Knee During Walking: The Use of the Approximate Entropy in Orthopaedics , 2006, Journal of Clinical Monitoring and Computing.

[65]  N. Wessel,et al.  Evaluation of renormalised entropy for risk stratification using heart rate variability data , 2000, Medical and Biological Engineering and Computing.

[66]  Li Shuangcheng,et al.  Measurement of climate complexity using sample entropy , 2006 .

[67]  P. Caminal,et al.  Compression entropy contributes to risk stratification in patients with cardiomyopathy / Kompressionsentropie zur verbesserten Risikostratifizierung bei Patienten mit DCM , 2006, Biomedizinische Technik. Biomedical engineering.

[68]  P. Ivanov.  Scale-invariant Aspects of Cardiac Dynamics Across Sleep Stages and Circadian Phases , 2007, 2006 International Conference of the IEEE Engineering in Medicine and Biology Society.

[69]  D. Abásolo,et al.  Entropy analysis of the EEG background activity in Alzheimer's disease patients , 2006, Physiological measurement.

[70]  M. Palaniswami,et al.  Journal of Neuroengineering and Rehabilitation Open Access a Comparative Study on Approximate Entropy Measure and Poincaré Plot Indexes of Minimum Foot Clearance Variability in the Elderly during Walking , 2008 .

[71]  Wangxin Yu,et al.  Characterization of Surface EMG Signal Based on Fuzzy Entropy , 2007, IEEE Transactions on Neural Systems and Rehabilitation Engineering.

[72]  Daniele Marinazzo,et al.  Multiscale analysis of short term heart beat interval, arterial blood pressure, and instantaneous lung volume time series , 2007, Artif. Intell. Medicine.

[73]  Daniel T. Schmitt,et al.  Fractal scale-invariant and nonlinear properties of cardiac dynamics remain stable with advanced age: a new mechanistic picture of cardiac control in healthy elderly. , 2007, American journal of physiology. Regulatory, integrative and comparative physiology.

[74]  H. Stanley,et al.  Power-law autocorrelated stochastic processes with long-range cross-correlations , 2007 .

[75]  K. Hlavácková-Schindler,et al.  Causality detection based on information-theoretic approaches in time series analysis , 2007 .

[76]  H. Stanley,et al.  Endogenous circadian rhythm in human motor activity uncoupled from circadian influences on cardiac dynamics , 2007, Proceedings of the National Academy of Sciences.

[77]  M. Masè,et al.  An integrated approach based on uniform quantization for the evaluation of complexity of short-term heart period variability: Application to 24 h Holter recordings in healthy and heart failure humans. , 2007, Chaos.

[78]  Stefano Panzeri,et al.  Correcting for the sampling bias problem in spike train information measures. , 2007, Journal of neurophysiology.

[79]  M. Struys,et al.  Behavior of Entropy/Complexity Measures of the Electroencephalogram during Propofol-induced Sedation: Dose-dependent Effects of Remifentanil , 2007, Anesthesiology.

[80]  A. Porta,et al.  Progressive decrease of heart period variability entropy-based complexity during graded head-up tilt. , 2007, Journal of applied physiology.

[81]  S. Pincus.  Approximate Entropy as an Irregularity Measure for Financial Data , 2008 .

[82]  Jeffrey M. Hausdorff,et al.  Levels of complexity in scale-invariant neural signals. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[83]  A. Seth,et al.  Granger causality and transfer entropy are equivalent for Gaussian variables. , 2009, Physical review letters.

[84]  Raúl Alcaraz,et al.  A review on sample entropy applications for the non-invasive analysis of atrial fibrillation electrocardiograms , 2010, Biomed. Signal Process. Control..

[85]  D. Cardinali,et al.  Nonlinear analysis of heart rate variability within independent frequency components during the sleep–wake cycle , 2010, Autonomic Neuroscience.

[86]  R. Motl,et al.  Real-life walking impairment in multiple sclerosis: preliminary comparison of four methods for processing accelerometry data , 2010, Multiple sclerosis.

[87]  S. Cerutti,et al.  Long-term Correlations and Complexity Analysis of the Heart Rate Variability Signal during Sleep , 2010, Methods of Information in Medicine.

[88]  Hong-Bo Xie,et al.  Fuzzy Approximate Entropy Analysis of Chaotic and Natural Complex Systems: Detecting Muscle Fatigue Using Electromyography Signals , 2010, Annals of Biomedical Engineering.

[89]  Danilo P Mandic,et al.  Multivariate multiscale entropy: a tool for complexity analysis of multichannel data. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[90]  Jochen Kaiser,et al.  Transfer entropy in magnetoencephalographic data: quantifying information flow in cortical and cerebellar networks. , 2011, Progress in biophysics and molecular biology.

[91]  N. Stergiou,et al.  Approximate entropy used to assess sitting postural sway of infants with developmental delay. , 2011, Infant behavior & development.

[92]  Breanna E. Studenka,et al.  Noise and Complexity in Human Postural Control: Interpreting the Different Estimations of Entropy , 2011, PloS one.

[93]  L. Faes,et al.  Information-based detection of nonlinear Granger causality in multivariate processes via a nonuniform embedding technique. , 2011, Physical review. E, Statistical, nonlinear, and soft matter physics.

[94]  N. Stergiou,et al.  Gait variability measures reveal differences between multiple sclerosis patients and healthy controls. , 2012, Motor control.

[95]  Mathias Baumert,et al.  Multiscale entropy and detrended fluctuation analysis of QT interval and heart rate variability during normal pregnancy , 2012, Comput. Biol. Medicine.

[96]  Albert Y. Zomaya,et al.  Local measures of information storage in complex distributed computation , 2012, Inf. Sci..

[97]  X. R. Wang,et al.  Quantifying and Tracing Information Cascades in Swarms , 2012, PloS one.

[98]  Jürgen Kurths,et al.  Escaping the curse of dimensionality in estimating multivariate transfer entropy. , 2012, Physical review letters.

[99]  A. Ledberg,et al.  Framework to study dynamic dependencies in networks of interacting processes. , 2012, Physical review. E, Statistical, nonlinear, and soft matter physics.

[100]  Dimitris Kugiumtzis,et al.  Direct coupling information measure from non-uniform embedding , 2013, Physical review. E, Statistical, nonlinear, and soft matter physics.

[101]  Jürgen Kurths,et al.  Statistical Mechanics and Information-Theoretic Perspectives on Complexity in the Earth System , 2013, Entropy.

[102]  L. Faes,et al.  Investigating the mechanisms of cardiovascular and cerebrovascular regulation in orthostatic syncope through an information decomposition strategy , 2013, Autonomic Neuroscience.

[103]  Niels Wessel,et al.  Practical considerations of permutation entropy , 2013, The European Physical Journal Special Topics.

[104]  Florentin Wörgötter,et al.  Information dynamics based self-adaptive reservoir for delay temporal memory tasks , 2013, Evol. Syst..

[105]  J. Detre,et al.  Brain Entropy Mapping Using fMRI , 2014, PloS one.

[106]  Joseph T. Lizier,et al.  Reduced predictable information in brain signals in autism spectrum disorder , 2014, Front. Neuroinform..

[107]  Dingchang Zheng,et al.  Assessing the complexity of short-term heartbeat interval series by distribution entropy , 2014, Medical & Biological Engineering & Computing.

[108]  P. Ivanov,et al.  Impact of Stock Market Structure on Intertrade Time and Price Dynamics , 2005, PloS one.

[109]  Bits from Biology for Computational Intelligence , 2014, ArXiv.

[110]  Karsten Keller,et al.  Ordinal Patterns, Entropy, and EEG , 2014, Entropy.

[111]  Moses O. Sokunbi,et al.  Sample entropy reveals high discriminative power between young and elderly adults in short fMRI data sets , 2014, Front. Neuroinform..

[112]  Anil K. Seth,et al.  The MVGC multivariate Granger causality toolbox: A new approach to Granger-causal inference , 2014, Journal of Neuroscience Methods.

[113]  L. Faes,et al.  Information dynamics of brain–heart physiological networks during sleep , 2014, New Journal of Physics.

[114]  Luca Faes,et al.  Estimating the decomposition of predictive information in multivariate systems. , 2015, Physical review. E, Statistical, nonlinear, and soft matter physics.

[115]  J. Sleigh,et al.  Permutation Lempel–Ziv complexity measure of electroencephalogram in GABAergic anaesthetics , 2015, Physiological measurement.

[116]  Huazhong Shu,et al.  Contribution to Transfer Entropy Estimation via the k-Nearest-Neighbors Approach , 2015, Entropy.

[117]  Luca Faes,et al.  Information Decomposition in Bivariate Systems: Theory and Application to Cardiorespiratory Dynamics , 2015, Entropy.

[118]  Viola Priesemann,et al.  Bits from Brains for Biologically Inspired Computing , 2014, Front. Robot. AI.

[119]  Luca Faes,et al.  Disentangling cardiovascular control mechanisms during head-down tilt via joint transfer entropy and self-entropy decompositions , 2015, Front. Physiol..

[120]  Jiang Wang,et al.  Characterization of complexity in the electroencephalograph activity of Alzheimer's disease based on fuzzy entropy. , 2015, Chaos.

[121]  A. Peters,et al.  Short-Term Heart Rate Variability—Influence of Gender and Age in Healthy Subjects , 2015, PloS one.

[122]  M. McAleer,et al.  An entropy-based analysis of the relationship between the DOW JONES Index and the TRNA Sentiment series , 2016 .

[123]  Luca Faes,et al.  Are Nonlinear Model-Free Conditional Entropy Approaches for the Assessment of Cardiac Control Complexity Superior to the Linear Model-Based One? , 2017, IEEE Transactions on Biomedical Engineering.

[124]  542 , 2019, Critical Care Medicine.

[125]  Rodríguez-Alvarez Lleretny,et al.  73 , 2019, Tao te Ching.