Combining Measures of Signal Complexity and Machine Learning for Time Series Analyis: A Review

Measures of signal complexity, such as the Hurst exponent, the fractal dimension, and the Spectrum of Lyapunov exponents, are used in time series analysis to give estimates on persistency, anti-persistency, fluctuations and predictability of the data under study. They have proven beneficial when doing time series prediction using machine and deep learning and tell what features may be relevant for predicting time-series and establishing complexity features. Further, the performance of machine learning approaches can be improved, taking into account the complexity of the data under study, e.g., adapting the employed algorithm to the inherent long-term memory of the data. In this article, we provide a review of complexity and entropy measures in combination with machine learning approaches. We give a comprehensive review of relevant publications, suggesting the use of fractal or complexity-measure concepts to improve existing machine or deep learning approaches. Additionally, we evaluate applications of these concepts and examine if they can be helpful in predicting and analyzing time series using machine and deep learning. Finally, we give a list of a total of six ways to combine machine learning and measures of signal complexity as found in the literature.

[1]  Petre Caraiani The predictive power of singular value decomposition entropy for stock market dynamics , 2014 .

[2]  Teuvo Kohonen,et al.  Self-Organizing Maps, Third Edition , 2001, Springer Series in Information Sciences.

[3]  Mostefa Mesbah,et al.  Range Entropy: A Bridge between Signal Complexity and Self-Similarity , 2018, Entropy.

[4]  Feller William,et al.  An Introduction To Probability Theory And Its Applications , 1950 .

[5]  S. J. Roberts,et al.  Temporal and spatial complexity measures for electroencephalogram based brain-computer interfacing , 2006, Medical & Biological Engineering & Computing.

[6]  Sebastian Raubitzek,et al.  A fractal interpolation approach to improve neural network predictions for difficult time series data , 2021, Expert Syst. Appl..

[7]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[8]  Fraser,et al.  Independent coordinates for strange attractors from mutual information. , 1986, Physical review. A, General physics.

[9]  S M Pincus,et al.  Irregularity and asynchrony in biologic network signals. , 2000, Methods in enzymology.

[10]  Concha Bielza,et al.  Machine Learning in Bioinformatics , 2008, Encyclopedia of Database Systems.

[11]  Aman Jantan,et al.  State-of-the-art in artificial neural network applications: A survey , 2018, Heliyon.

[12]  J. Richman,et al.  Physiological time-series analysis using approximate entropy and sample entropy. , 2000, American journal of physiology. Heart and circulatory physiology.

[13]  C. Tan,et al.  NEURAL NETWORKS FOR TECHNICAL ANALYSIS: A STUDY ON KLCI , 1999 .

[14]  John A. Belward,et al.  Fractal dimensions for rainfall time series , 1999 .

[15]  Q. Fu,et al.  Analysis of Irrigation Water Use Efficiency Based on the Chaos Features of a Rainfall Time Series , 2017, Water Resources Management.

[16]  Khan Muhammad,et al.  Characterizing Complexity and Self-Similarity Based on Fractal and Entropy Analyses for Stock Market Forecast Modelling , 2020, Expert Syst. Appl..

[17]  T. Neubauer,et al.  Taming the Chaos in Neural Network Time Series Predictions , 2021, Entropy.

[18]  Z. Vörös On multifractality of high-latitude geomagnetic fluctuations , 2000 .

[19]  P. Devynck,et al.  The Hurst exponent and long-time correlation , 2000 .

[20]  Nikola K. Kasabov,et al.  DENFIS: dynamic evolving neural-fuzzy inference system and its application for time-series prediction , 2002, IEEE Trans. Fuzzy Syst..

[21]  Leo Breiman,et al.  Random Forests , 2001, Machine Learning.

[22]  Xuelong Li,et al.  Multivariate Multilinear Regression , 2012, IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics).

[23]  Woojin Chang,et al.  Predicting the Direction of US Stock Prices Using Effective Transfer Entropy and Machine Learning Techniques , 2020, IEEE Access.

[24]  Michael Kirley,et al.  Predicting Stock Market Time Series Using Evolutionary Artificial Neural Networks with Hurst Exponent Input Windows , 2006, Australian Conference on Artificial Intelligence.

[25]  A. Wolf,et al.  Determining Lyapunov exponents from a time series , 1985 .

[26]  J. R. Wallis,et al.  Noah, Joseph, and Operational Hydrology , 1968 .

[27]  M. J. Katz,et al.  Fractals and the analysis of waveforms. , 1988, Computers in biology and medicine.

[28]  R. Gomory,et al.  Fractals and Scaling in Finance: Discontinuity, Concentration, Risk. Selecta Volume E , 1997 .

[29]  Yoshua Bengio,et al.  Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation , 2014, EMNLP.

[30]  E. Basar,et al.  Wavelet entropy: a new tool for analysis of short duration brain electrical signals , 2001, Journal of Neuroscience Methods.

[31]  Brown Calculating Lyapunov exponents for short and/or noisy data sets. , 1993, Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topics.

[32]  J. L. Hodges,et al.  Discriminatory Analysis - Nonparametric Discrimination: Consistency Properties , 1989 .

[33]  Àngela Nebot,et al.  Hybrid methodologies for electricity load forecasting: Entropy-based feature selection with machine learning and soft computing techniques , 2015 .

[34]  Michal Veselý,et al.  Entropy of fractal systems , 2013, Comput. Math. Appl..

[35]  Yanmin Shao,et al.  How long the singular value decomposed entropy predicts the stock market? — Evidence from the Dow Jones Industrial Average Index , 2016 .

[36]  Schreiber,et al.  Measuring information transfer , 2000, Physical review letters.

[37]  酒井 憲司 Nonlinear dynamics and chaos in agricultural systems , 2001 .

[38]  Anil K. Jain,et al.  Artificial Neural Networks: A Tutorial , 1996, Computer.

[39]  Hasbi Yasin,et al.  Cascade Forward Neural Network for Time Series Prediction , 2018 .

[40]  D. Cox The Regression Analysis of Binary Sequences , 1958 .

[41]  Sebastian Raubitzek,et al.  Machine Learning and Chaos Theory in Agriculture , 2020, ERCIM News.

[42]  Dumitru Baleanu,et al.  A NOVEL R/S FRACTAL ANALYSIS AND WAVELET ENTROPY CHARACTERIZATION APPROACH FOR ROBUST FORECASTING BASED ON SELF-SIMILAR TIME SERIES MODELING , 2020 .

[43]  A. Laub,et al.  The singular value decomposition: Its computation and some applications , 1980 .

[44]  Robert König,et al.  The Operational Meaning of Min- and Max-Entropy , 2008, IEEE Transactions on Information Theory.

[45]  Zhi-Wei Ni,et al.  Stock trend prediction based on fractal feature selection and support vector machine , 2011, Expert Syst. Appl..

[46]  Chen Liqun,et al.  On properties of hyperchaos: Case study , 1999 .

[47]  M. Marques,et al.  Recent advances and applications of machine learning in solid-state materials science , 2019, npj Computational Materials.

[48]  J. Friedman Greedy function approximation: A gradient boosting machine. , 2001 .

[49]  Yasuhiko Dote,et al.  Novel time series analysis and prediction of stock trading using fractal theory and time delayed neural network , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[50]  Alejandra Figliola,et al.  Time-frequency analysis of electroencephalogram series. III. Wavelet packets and information cost function , 1998 .

[51]  Audrey L. Mayer,et al.  Fisher Information and dynamic regime changes in ecological systems , 2006 .

[52]  Alexander Marshak,et al.  Approximate Entropy and Sample Entropy: A Comprehensive Tutorial , 2019, Entropy.

[53]  Luiz Paulo Lopes Fávero,et al.  Hurst exponent, fractals and neural networks for forecasting financial asset returns in Brazil , 2018 .

[54]  Arthur Petrosian,et al.  Kolmogorov complexity of finite sequences and recognition of different preictal EEG patterns , 1995, Proceedings Eighth IEEE Symposium on Computer-Based Medical Systems.

[55]  T. D. Matteo,et al.  Multi-scaling in finance , 2007 .

[56]  Tianqi Chen,et al.  XGBoost: A Scalable Tree Boosting System , 2016, KDD.

[57]  Bush Jones,et al.  Architecture of systems problem solving , 1986, Journal of the American Society for Information Science.

[58]  A. Krakovská,et al.  Use of False Nearest Neighbours for Selecting Variables and Embedding Parameters for State Space Reconstruction , 2015 .

[59]  N. Sopipan,et al.  Forecasting SET50 Index with Multiple Regression based on Principal Component Analysis , 2012 .

[60]  Khaled Rasheed,et al.  Stock market prediction with multiple classifiers , 2007, Applied Intelligence.

[61]  B. Mandelbrot Fractals and Scaling In Finance: Discontinuity, Concentration, Risk , 2010 .

[62]  K. Rasheed,et al.  HURST EXPONENT AND FINANCIAL MARKET PREDICTABILITY , 2005 .

[63]  Oscar Castillo,et al.  Hybrid intelligent systems for time series prediction using neural networks, fuzzy logic, and fractal theory , 2002, IEEE Trans. Neural Networks.

[64]  Approximate , 2020, Definitions.

[65]  Indranil Ghosh,et al.  Fractal Investigation and Maximal Overlap Discrete Wavelet Transformation (MODWT)-based Machine Learning Framework for Forecasting Exchange Rates , 2017 .

[66]  Pritpal Singh,et al.  A brief review of modeling approaches based on fuzzy time series , 2015, International Journal of Machine Learning and Cybernetics.

[67]  Indranil Ghosh,et al.  Fractal Inspection and Machine Learning-Based Predictive Modelling Framework for Financial Markets , 2018 .

[68]  S M Pincus,et al.  Approximate entropy as a measure of system complexity. , 1991, Proceedings of the National Academy of Sciences of the United States of America.

[69]  R. Khanna,et al.  Support Vector Regression , 2015 .

[70]  Sawada,et al.  Measurement of the Lyapunov spectrum from a chaotic time series. , 1985, Physical review letters.

[71]  M. Rosenstein,et al.  A practical method for calculating largest Lyapunov exponents from small data sets , 1993 .

[72]  J. Friedrich,et al.  Stochastic Interpolation of Sparsely Sampled Time Series via Multipoint Fractional Brownian Bridges. , 2020, Physical review letters.

[73]  FRACTALS AND THEIR APPLICABILITY IN GEOLOGICAL WIRELINE LOG ANALYSIS , 1996 .

[74]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[75]  Nicholas G. Polson,et al.  Deep Learning in Finance , 2016, ArXiv.

[76]  Z. Voros,et al.  Neural network prediction of geomagnetic activity: a method using local H\"{o}lder exponents , 2004, physics/0411062.

[77]  Michael I. Jordan Attractor dynamics and parallelism in a connectionist sequential machine , 1990 .

[78]  Johannes Fürnkranz,et al.  Decision Tree , 2010, Encyclopedia of Machine Learning and Data Mining.

[79]  Ahmed Tealab,et al.  Time series forecasting using artificial neural networks methodologies: A systematic review , 2018, Future Computing and Informatics Journal.

[80]  Salah Sukkarieh,et al.  Machine learning approaches for crop yield prediction and nitrogen status estimation in precision agriculture: A review , 2018, Comput. Electron. Agric..

[81]  T. Higuchi Approach to an irregular time series on the basis of the fractal theory , 1988 .

[82]  E. Mizutani,et al.  Neuro-Fuzzy and Soft Computing-A Computational Approach to Learning and Machine Intelligence [Book Review] , 1997, IEEE Transactions on Automatic Control.

[83]  T. Di Matteo Multi-scaling in finance , 2007 .

[84]  Nello Cristianini,et al.  An Introduction to Support Vector Machines and Other Kernel-based Learning Methods , 2000 .

[85]  Luísa Castro,et al.  Nonlinear Methods Most Applied to Heart-Rate Time Series: A Review , 2020, Entropy.

[86]  R. Eykholt,et al.  Estimating the Lyapunov-exponent spectrum from short time series of low precision. , 1991, Physical review letters.

[87]  Soteris A. Kalogirou,et al.  Machine learning methods for solar radiation forecasting: A review , 2017 .

[88]  Jensen,et al.  Fractal measures and their singularities: The characterization of strange sets. , 1987, Physical review. A, General physics.

[89]  Eckmann,et al.  Liapunov exponents from time series. , 1986, Physical review. A, General physics.

[90]  K. Briggs An improved method for estimating Liapunov exponents of chaotic time series , 1990 .

[91]  Yu-Chi Ho Neuro-fuzzy And Soft Computing - A Computational Approach To Learning And Machine Intelligence [Book Reviews] , 1998, Proceedings of the IEEE.

[92]  Jason Brownlee,et al.  Clever Algorithms: Nature-Inspired Programming Recipes , 2012 .

[93]  Paul S Addison,et al.  Wavelet transforms and the ECG: a review , 2005, Physiological measurement.

[94]  K. Borgwardt,et al.  Machine Learning in Medicine , 2015, Mach. Learn. under Resour. Constraints Vol. 3.

[95]  Henry J. Kelley,et al.  Gradient Theory of Optimal Flight Paths , 1960 .

[96]  H. Kantz,et al.  Analysing the information flow between financial time series , 2002 .

[97]  Bernhard E. Boser,et al.  A training algorithm for optimal margin classifiers , 1992, COLT '92.

[98]  Eugen Diaconescu,et al.  The use of NARX neural networks to predict chaotic time series , 2008 .

[99]  J. Elsner,et al.  Chaos, Strange Attractors, and Weather. , 1989 .

[100]  Ping Wang,et al.  Fuzzy Forecasting with Fractal Analysis for the Time Series of Environmental Pollution , 2013, Time Series Analysis, Modeling and Applications.