Parallel Asynchronous Strategies for the Execution of Feature Selection Algorithms

Reducing the dimensionality of datasets is a fundamental step in the task of building a classification model. Feature selection is the process of selecting a smaller subset of features from the original one in order to enhance the performance of the classification model. The problem is known to be NP-hard, and despite the existence of several algorithms there is not one that outperforms the others in all scenarios. Due to the complexity of the problem usually feature selection algorithms have to compromise the quality of their solutions in order to execute in a practicable amount of time. Parallel computing techniques emerge as a potential solution to tackle this problem. There are several approaches that already execute feature selection in parallel resorting to synchronous models. These are preferred due to their simplicity and capability to use with any feature selection algorithm. However, synchronous models implement pausing points during the execution flow, which decrease the parallel performance. In this paper, we discuss the challenges of executing feature selection algorithms in parallel using asynchronous models, and present a feature selection algorithm that favours these models. Furthermore, we present two strategies for an asynchronous parallel execution not only of our algorithm but of any other feature selection approach. The first strategy solves the problem using the distributed memory paradigm, while the second exploits the use of shared memory. We evaluate the parallel performance of our strategies using up to 32 cores. The results show near linear speedups for both strategies, with the shared memory strategy outperforming the distributed one. Additionally, we provide an example of adapting our strategies to execute the Sequential forward Search asynchronously. We further test this version versus a synchronous one. Our results revealed that, by using an asynchronous strategy, we are able to save an average of 7.5% of the execution time.

[1]  Yafei Zhang,et al.  Dynamic Adaboost learning with feature selection based on parallel genetic algorithm for image annotation , 2010, Knowl. Based Syst..

[2]  Sukumar Bandopadhyay,et al.  An Objective Analysis of Support Vector Machine Based Classification for Remote Sensing , 2008 .

[3]  Huan Liu,et al.  Feature selection for classification: A review , 2014 .

[4]  Hiroshi Sako,et al.  Comparison of genetic algorithm and sequential search methods for classifier subset selection , 2003, Seventh International Conference on Document Analysis and Recognition, 2003. Proceedings..

[5]  Huan Liu,et al.  Feature Selection for Classification , 1997, Intell. Data Anal..

[6]  Leslie G. Valiant,et al.  Direct Bulk-Synchronous Parallel Algorithms , 1992, J. Parallel Distributed Comput..

[7]  Helena Moniz,et al.  Speech Features for Discriminating Stress Using Branch and Bound Wrapper Search , 2015, SLATE.

[8]  Josef Kittler,et al.  Fast branch & bound algorithms for optimal feature selection , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[9]  Verónica Bolón-Canedo,et al.  Toward parallel feature selection from vertically partitioned data , 2014, ESANN.

[10]  S. G. Djorgovski,et al.  Feature selection strategies for classifying high dimensional astronomical data sets , 2013, 2013 IEEE International Conference on Big Data.

[11]  Jennifer G. Dy,et al.  GPU-Accelerated Feature Selection for Outlier Detection Using the Local Kernel Density Ratio , 2012, 2012 IEEE 12th International Conference on Data Mining.

[12]  Huan Liu,et al.  Toward integrating feature selection algorithms for classification and clustering , 2005, IEEE Transactions on Knowledge and Data Engineering.

[13]  Stan Matwin,et al.  Parallelizing Feature Selection , 2006, Algorithmica.

[14]  Yves Robert,et al.  The master-slave paradigm with heterogeneous processors , 2001, Proceedings 42nd IEEE Symposium on Foundations of Computer Science.

[15]  John Loughrey,et al.  Using Early-Stopping to Avoid Overfitting in Wrapper-Based Feature Selection Employing Stochastic Search , 2005 .

[16]  Steve R. Gunn,et al.  Result Analysis of the NIPS 2003 Feature Selection Challenge , 2004, NIPS.

[17]  AlbaEnrique,et al.  Analyzing synchronous and asynchronous parallel distributed genetic algorithms , 2001 .

[18]  P. Stenstrom.  A survey of cache coherence schemes for multiprocessors , 1990, Computer.

[19]  Pedro M. Domingos.  A few useful things to know about machine learning , 2012, Commun. ACM.

[20]  Zheng Zhao,et al.  Massively parallel feature selection: an approach based on variance preservation , 2012, Machine Learning.

[21]  Jeremy Kubica,et al.  Scaling Up Machine Learning: Parallel Large-Scale Feature Selection , 2011 .

[22]  Fernando M. A. Silva,et al.  A Parallel Computing Hybrid Approach for Feature Selection , 2015, 2015 IEEE 18th International Conference on Computational Science and Engineering.

[23]  Wilfried N. Gansterer,et al.  On the Relationship Between Feature Selection and Classification Accuracy , 2008, FSDM.

[24]  Lluís A. Belanche Muñoz,et al.  Feature selection algorithms: a survey and experimental evaluation , 2002, 2002 IEEE International Conference on Data Mining, 2002. Proceedings..

[25]  Enrique Alba,et al.  Analyzing synchronous and asynchronous parallel distributed genetic algorithms , 2001, Future Gener. Comput. Syst..

[26]  Venu Govindaraju,et al.  Parallel Feature Selection Inspired by Group Testing , 2014, NIPS.