Learning symbolic representations of hybrid dynamical systems

A hybrid dynamical system is a mathematical model suitable for describing an extensive spectrum of multi-modal, time-series behaviors, ranging from bouncing balls to air traffic controllers. This paper describes multi-modal symbolic regression (MMSR): a learning algorithm to construct non-linear symbolic representations of discrete dynamical systems with continuous mappings from unlabeled, time-series data. MMSR consists of two subalgorithms--clustered symbolic regression, a method to simultaneously identify distinct behaviors while formulating their mathematical expressions, and transition modeling, an algorithm to infer symbolic inequalities that describe binary classification boundaries. These subalgorithms are combined to infer hybrid dynamical systems as a collection of apt, mathematical expressions. MMSR is evaluated on a collection of four synthetic data sets and outperforms other multi-modal machine learning approaches in both accuracy and interpretability, even in the presence of noise. Furthermore, the versatility of MMSR is demonstrated by identifying and inferring classical expressions of transistor modes from recorded measurements.

[1]  Ray J. Solomonoff,et al.  A Formal Theory of Inductive Inference. Part II , 1964, Inf. Control..

[2]  H. Akaike A new look at the statistical model identification , 1974 .

[3]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[4]  A. Sedra Microelectronic circuits , 1982 .

[5]  A.M. Davis,et al.  Microelectronic circuits , 1983, Proceedings of the IEEE.

[6]  Nichael Lynn Cramer,et al.  A Representation for the Adaptive Generation of Simple Sequential Programs , 1985, ICGA.

[7]  Geoffrey E. Hinton,et al.  Learning representations by back-propagating errors , 1986, Nature.

[8]  Geoffrey E. Hinton,et al.  Learning representations by back-propagation errors, nature , 1986 .

[9]  Geoffrey E. Hinton,et al.  Adaptive Mixtures of Local Experts , 1991, Neural Computation.

[10]  Sheng Chen,et al.  Recursive hybrid algorithm for non-linear system identification using radial basis function networks , 1992 .

[11]  John R. Koza,et al.  Genetic programming - on the programming of computers by means of natural selection , 1993, Complex adaptive systems.

[12]  C. Lee Giles,et al.  Extraction, Insertion and Refinement of Symbolic Rules in Dynamically Driven Recurrent Neural Networks , 1993 .

[13]  Yoshua Bengio,et al.  An Input Output HMM Architecture , 1994, NIPS.

[14]  Roland Olsson,et al.  Inductive Functional Programming Using Incremental Program Transformation , 1995, Artif. Intell..

[15]  Marios M. Polycarpou,et al.  High-order neural network structures for identification of dynamical systems , 1995, IEEE Trans. Neural Networks.

[16]  C. Lee Giles,et al.  Constructing deterministic finite-state automata in recurrent neural networks , 1996, JACM.

[17]  C. Lee Giles,et al.  Extraction of rules from discrete-time recurrent neural networks , 1996, Neural Networks.

[18]  Don R. Hush,et al.  Bounds on the complexity of recurrent neural network implementations of finite state machines , 1993, Neural Networks.

[19]  Thomas A. Henzinger,et al.  The theory of hybrid automata , 1996, Proceedings 11th Annual IEEE Symposium on Logic in Computer Science.

[20]  S. Shankar Sastry,et al.  Conflict resolution for air traffic management: a study in multiagent hybrid systems , 1998, IEEE Trans. Autom. Control..

[21]  David L. Dowe,et al.  Minimum Message Length and Kolmogorov Complexity , 1999, Comput. J..

[22]  Ilona Jagielska,et al.  An investigation into the application of neural networks, fuzzy logic, genetic algorithms, and rough sets to automated knowledge acquisition for classification problems , 1999, Neurocomputing.

[23]  Johannes Schumacher,et al.  An Introduction to Hybrid Dynamical Systems, Springer Lecture Notes in Control and Information Sciences 251 , 1999 .

[24]  Sébastien Marcel,et al.  Hand gesture recognition using input-output hidden Markov models , 2000, Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (Cat. No. PR00580).

[25]  V. Sanguineti,et al.  Connecting Brains to Robots: The Development of a Hybrid System for the Study of Learning in Neural Tissues , 2000 .

[26]  John S. McCaskill,et al.  Connecting Brains to Robots: The Development of a Hybrid System for the Study of Learning in Neural Tissues , 2000 .

[27]  Jordi Vitrià,et al.  Learning mixture models using a genetic version of the EM algorithm , 2000, Pattern Recognition Letters.

[28]  Alex Alves Freitas,et al.  Discovering Fuzzy Classification Rules with Genetic Programming and Co-evolution , 2001, PKDD.

[29]  Mehmet Kaya,et al.  Determination of fuzzy logic membership functions using genetic algorithms , 2001, Fuzzy Sets Syst..

[30]  Leo Breiman,et al.  Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author) , 2001, Statistical Science.

[31]  Hitoshi Iba,et al.  Inference of differential equation models by genetic programming , 2002, Inf. Sci..

[32]  Jan Lunze,et al.  What Is a Hybrid System , 2002 .

[33]  Manfred Morari,et al.  A clustering technique for the identification of piecewise affine systems , 2001, Autom..

[34]  S. Sastry,et al.  An algebraic geometric approach to the identification of a class of linear hybrid systems , 2003, 42nd IEEE International Conference on Decision and Control (IEEE Cat. No.03CH37475).

[35]  Christian W. Omlin,et al.  A Machine Learning Method for Extracting Symbolic Knowledge from Recurrent Neural Networks , 2004, Neural Computation.

[36]  Nikhil R. Pal,et al.  A novel approach to design classifiers using genetic programming , 2004, IEEE Transactions on Evolutionary Computation.

[37]  A. Anderson,et al.  A hybrid mathematical model of solid tumour invasion: the importance of cell adhesion. , 2005, Mathematical medicine and biology : a journal of the IMA.

[38]  A.M. Gonzalez,et al.  Modeling and forecasting electricity prices with input/output hidden Markov models , 2005, IEEE Transactions on Power Systems.

[39]  Michael S. Branicky,et al.  Introduction to Hybrid Systems , 2005, Handbook of Networked and Embedded Control Systems.

[40]  John Baillieul,et al.  Handbook of Networked and Embedded Control Systems , 2005, Handbook of Networked and Embedded Control Systems.

[41]  Henrik Jacobsson,et al.  Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review , 2005, Neural Computation.

[42]  Djamel Bouchaffra,et al.  Genetic-based EM algorithm for learning Gaussian mixture models , 2005, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Christopher M. Bishop,et al.  Pattern Recognition and Machine Learning (Information Science and Statistics) , 2006 .

[44]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[45]  William S. Levine,et al.  Handbook Of Networked And Embedded Control Systems , 2007 .

[46]  René Vidal,et al.  Identification of Hybrid Systems: A Tutorial , 2007, Eur. J. Control.

[47]  Hod Lipson,et al.  Distilling Free-Form Natural Laws from Experimental Data , 2009, Science.

[48]  Dick den Hertog,et al.  Order of Nonlinearity as a Complexity Measure for Models Generated by Symbolic Regression via Pareto Genetic Programming , 2009, IEEE Transactions on Evolutionary Computation.

[49]  Michael D. Schmidt,et al.  Symbolic Regression of Implicit Equations , 2010 .

[50]  Gérard Bloch,et al.  Reduced-Size Kernel Models for Nonlinear Hybrid System Identification , 2011, IEEE Transactions on Neural Networks.

[51]  J. Eisert,et al.  Extracting dynamical equations from experimental data is NP hard. , 2010, Physical review letters.