Towards Automatic Composition of Multicomponent Predictive Systems

Automatic composition and parametrisation of multicomponent predictive systems (MCPSs) consisting of chains of data transformation steps is a challenging task. In this paper we propose and describe an extension to the Auto-WEKA software which now allows to compose and optimise such flexible MCPSs by using a sequence of WEKA methods. In the experimental analysis we focus on examining the impact of significantly extending the search space by incorporating additional hyperparameters of the models, on the quality of the found solutions. In a range of extensive experiments three different optimisation strategies are used to automatically compose MCPSs on 21 publicly available datasets. A comparison with previous work indicates that extending the search space improves the classification accuracy in the majority of the cases. The diversity of the found MCPSs are also an indication that fully and automatically exploiting different combinations of data cleaning and preprocessing techniques is possible and highly beneficial for different predictive models. This can have a big impact on high quality predictive models development, maintenance and scalability aspects needed in modern application and deployment scenarios.

[1]  Thierry J. Chaussalet,et al.  Data preparation for clinical data mining to identify patients at risk of readmission , 2010, 2010 IEEE 23rd International Symposium on Computer-Based Medical Systems (CBMS).

[2]  Bogdan Gabrys,et al.  Meta-learning for time series forecasting and forecast combination , 2010, Neurocomputing.

[3]  Bogdan Gabrys,et al.  Multicriteria approaches for predictive model generation: A comparative experimental study , 2014, 2014 IEEE Symposium on Computational Intelligence in Multi-Criteria Decision-Making (MCDM).

[4]  Dorian Pyle,et al.  Data Preparation for Data Mining , 1999 .

[5]  Kevin Leyton-Brown,et al.  Sequential Model-Based Optimization for General Algorithm Configuration , 2011, LION.

[6]  Frank Hutter,et al.  Using Meta-Learning to Initialize Bayesian Optimization of Hyperparameters , 2014, MetaSel@ECAI.

[7]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[8]  Michael J. A. Berry,et al.  Data Mining Techniques: For Marketing, Sales, and Customer Relationship Management , 2004 .

[9]  Dong Zhou,et al.  Translation techniques in cross-language information retrieval , 2012, CSUR.

[10]  X. C. Guo,et al.  A novel LS-SVMs hyper-parameter selection based on particle swarm optimization , 2008, Neurocomputing.

[11]  Jasper Snoek,et al.  Multi-Task Bayesian Optimization , 2013, NIPS.

[12]  A. McQuarrie,et al.  Regression and Time Series Model Selection , 1998 .

[13]  Katharina Eggensperger,et al.  Towards an Empirical Foundation for Assessing Bayesian Optimization of Hyperparameters , 2013 .

[14]  Nando de Freitas,et al.  A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning , 2010, ArXiv.

[15]  Abraham Bernstein,et al.  A survey of intelligent assistants for data analysis , 2013, CSUR.

[16]  Volker Märgner,et al.  A design of a preprocessing framework for large database of historical documents , 2011, HIP '11.

[17]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[18]  Athanasios Tsakonas,et al.  From Sensor Readings to Predictions: On the Process of Developing Practical Soft Sensors , 2014, IDA.

[19]  Joaquin Vanschoren,et al.  Selecting Classification Algorithms with Active Testing , 2012, MLDM.

[20]  Bogdan Gabrys,et al.  Density-Preserving Sampling: Robust and Efficient Alternative to Cross-Validation for Error Estimation , 2013, IEEE Transactions on Neural Networks and Learning Systems.

[21]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[22]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[23]  Kevin Leyton-Brown,et al.  Efficient Benchmarking of Hyperparameter Optimizers via Surrogates , 2015, AAAI.

[24]  Ting Wang,et al.  A general framework for medical data mining , 2010, 2010 International Conference on Future Information Technology and Management Engineering.

[25]  Yoshua Bengio,et al.  Gradient-Based Optimization of Hyperparameters , 2000, Neural Computation.