Towards a Benchmark for Evaluating Design Pattern Miner Tools

Recovering design pattern usage in source code is a very difficult task. Several tools are described in the literature for this purpose, but there is little work invested in evaluating them. The main reason for this is the lack of an approved benchmark for these tools. In this paper we present work in progress towards creating a benchmark, called DEEBEE (design pattern evaluation benchmark environment), for evaluating and comparing design pattern miner tools. It is programming language, tool, pattern and software independent, and it is open to the community and freely available. Currently, the benchmark database contains the results of three tools: Columbus (C++), Maisa (C++), and design pattern detection tool (Java). The tools were evaluated on reference implementations of patterns and on open source software (Mozilla, NotePad++, JHotDraw, JRefactory and JUnit). Additionally, instances recovered by researchers are added from NotePad++ as well. Some recovered patterns are already verified by experienced developers. This work is the first step in building a large reference database of design pattern usage in open source software and we expect that researchers will join us in this effort.

[1]  Giuliano Antoniol,et al.  Object oriented design pattern inference , 1999, Proceedings IEEE International Conference on Software Maintenance - 1999 (ICSM'99). 'Software Maintenance for Business Change' (Cat. No.99CB36360).

[2]  Yann-Gaël Guéhéneuc,et al.  Instantiating and detecting design patterns: putting bits and pieces together , 2001, Proceedings 16th Annual International Conference on Automated Software Engineering (ASE 2001).

[3]  Yann-Gaël Guéhéneuc.  Using explanations for design patterns identification , 2001 .

[4]  Joakim Nivre,et al.  On Evaluation of Accuracy in Pattern Detection , 2006 .

[5]  Yann-Gaël Guéhéneuc,et al.  A comparative framework for design recovery tools , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[6]  Rudolf Ferenc,et al.  Evaluating C++ Design Pattern Miner Tools , 2006, 2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation.

[7]  Joakim Nivre,et al.  Evaluation of Accuracy in Design Pattern Occurrence Detection , 2010, IEEE Transactions on Software Engineering.

[8]  Alexander Chatzigeorgiou,et al.  Design Pattern Detection Using Similarity Scoring , 2006, IEEE Transactions on Software Engineering.

[9]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[10]  Anssi Karhinen,et al.  Software Metrics by Architectural Pattern Mining , 2000 .

[11]  Giuliano Antoniol,et al.  Using metrics to identify design patterns in object-oriented software , 1998, Proceedings Fifth International Software Metrics Symposium. Metrics (Cat. No.98TB100262).

[12]  Rudolf Ferenc,et al.  Mining design patterns from C++ source code , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[13]  Dirk Beyer,et al.  CrocoPat: efficient pattern analysis in object-oriented programs , 2003, 11th IEEE International Workshop on Program Comprehension, 2003..

[14]  Susan Elliott Sim,et al.  A structured demonstration of five program comprehension tools: lessons learnt , 2000, Proceedings Seventh Working Conference on Reverse Engineering.

[15]  Susan Elliott Sim,et al.  A structured demonstration of program comprehension tools , 2000, Proceedings Seventh Working Conference on Reverse Engineering.

[16]  Giuliano Antoniol,et al.  Comparison and Evaluation of Clone Detection Tools , 2007, IEEE Transactions on Software Engineering.

[17]  Michele Risi,et al.  Design pattern recovery by visual language parsing , 2005, Ninth European Conference on Software Maintenance and Reengineering.

[18]  Susan Elliott Sim,et al.  On using a benchmark to evaluate C++ extractors , 2002, Proceedings 10th International Workshop on Program Comprehension.

[19]  Tibor Gyimóthy,et al.  Columbus - reverse engineering tool and schema for C++ , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[20]  Lutz Prechelt,et al.  Design recovery by automated search for structural design patterns in object-oriented software , 1996, Proceedings of WCRE '96: 4rd Working Conference on Reverse Engineering.

[21]  Susan Elliott Sim,et al.  Using benchmarking to advance research: a challenge to software engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[22]  Susan Elliott Sim,et al.  A theory of benchmarking with applications to software reverse engineering , 2003 .

[23]  Yann-Gaël Guéhéneuc,et al.  Efficient identification of design patterns with bit-vector algorithm , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).