Developing Genetic Algorithms Using Different MapReduce Frameworks: MPI vs. Hadoop

MapReduce is a quite popular paradigm, which allows to no specialized users to use large parallel computational platforms in a transparent way. Hadoop is the most used implementation of this paradigm, and in fact, for a large amount of users the word Hadoop and MapReduce are interchangeable. But, there are other frameworks that implement this programming paradigm, such as MapReduce-MPI. Since, optimization techniques can be greatly beneficiary of this kind of data-intensive computing modeling, in this paper, we analyze the performance effect of developing genetic algorithms (GA) using different frameworks of MapReduce (MRGA). In particular, we implement MRGA using Hadoop and MR-MPI frameworks. We analyze and compare both implementations considering relevant aspects such as efficiency and scalability to solve a large dimension problem. The results show a similar efficiency level between the algorithms but Hadoop presents a better scalability.

[1]  Xavier Llorà,et al.  Scaling Genetic Algorithms Using MapReduce , 2009, 2009 Ninth International Conference on Intelligent Systems Design and Applications.

[2]  Linda Di Geronimo,et al.  A Parallel Genetic Algorithm Based on Hadoop MapReduce for the Automatic Generation of JUnit Test Suites , 2012, 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation.

[3]  Filomena Ferrucci,et al.  Using Hadoop MapReduce for Parallel Genetic Algorithms: A Comparison of the Global, Grid and Island Models , 2018, Evolutionary Computation.

[4]  Sanjay Ghemawat,et al.  MapReduce: Simplified Data Processing on Large Clusters , 2004, OSDI.

[5]  Steven J. Plimpton,et al.  MapReduce in MPI for Large-scale graph algorithms , 2011, Parallel Comput..

[6]  David Pisinger,et al.  Core Problems in Knapsack Algorithms , 1999, Oper. Res..

[7]  David S. Johnson,et al.  Computers and Intractability: A Guide to the Theory of NP-Completeness , 1978 .

[8]  David E. Goldberg,et al.  The Design of Innovation: Lessons from and for Competent Genetic Algorithms , 2002 .

[9]  Xavier Llorà,et al.  Scaling eCGA model building via data-intensive computing , 2010, IEEE Congress on Evolutionary Computation.

[10]  El-Ghazali Talbi,et al.  Metaheuristics - From Design to Implementation , 2009 .

[11]  David E. Goldberg,et al.  Genetic Algorithms, Tournament Selection, and the Effects of Noise , 1995, Complex Syst..

[12]  Sebastián Ventura,et al.  Extremely high-dimensional optimization with MapReduce: Scaling functions and algorithm , 2017, Inf. Sci..