State-of-the-Art and Evolution in Public Data Sets and Competitions for System Identification, Time Series Prediction and Pattern Recognition

It is the aim of reproducible research to provide mechanisms for objective comparison of methods, algorithms, software and procedures in various research topics. In this paper, we discuss the role of data sets, benchmarks and competitions in the fields of system identification, time series prediction, classification, and pattern recognition in view of creating an environment of reproducible research. Important elements are the data sets, their origin, and the comparison measures that will be used to rank the performance of the methods. The issues are discussed, a comparison is made and recommendations are given.

[1]  Andreas S. Weigend,et al.  Time Series Prediction: Forecasting the Future and Understanding the Past , 1994 .

[2]  Bart De Schutter,et al.  DAISY : A database for identification of systems , 1997 .

[3]  Johan A. K. Suykens,et al.  The K.U.Leuven Time Series Prediction Competition , 1998 .

[4]  James McNames,et al.  A Nearest Trajectory Strategy for Time Series Prediction , 2000 .

[5]  Ming-Wei Chang,et al.  EUNITE Network Competition: Electricity Load Forecasting , 2001 .

[6]  Paul Jung,et al.  No free lunch. , 2002, Health affairs.

[7]  Johan A. K. Suykens,et al.  Systematic benchmarking of microarray data classification: assessing the role of non-linearity and dimensionality reduction , 2004, Bioinform..

[8]  Johan A. K. Suykens,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2004, Machine Learning.

[9]  Jouko Lampinen,et al.  Time series prediction by Kalman smoother with cross-validated noise density , 2004, 2004 IEEE International Joint Conference on Neural Networks (IEEE Cat. No.04CH37541).

[10]  Ming-Wei Chang,et al.  Load Forecasting Using Support Vector Machines: A Study on EUNITE Competition 2001 , 2004, IEEE Transactions on Power Systems.

[11]  K. U. Leuven.  A COMPARATIVE STUDY OF LS-SVM’S APPLIED TO THE SILVER BOX IDENTIFICATION PROBLEM , 2004 .

[12]  J. Suykens,et al.  A comparative study of ls-svm’s applied to the silver box identification problem , 2004 .

[13]  C. Sunstein.  Infotopia: How Many Minds Produce Knowledge , 2006 .

[14]  J. Buhmann,et al.  Performance Prediction Challenge , 2006, The 2006 IEEE International Joint Conference on Neural Network Proceedings.

[15]  Bladimir Díaz Borges.  Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities , 2008 .

[16]  K. Johana,et al.  Benchmarking Least Squares Support Vector Machine Classifiers , 2022 .