Evaluation of Accuracy in Design Pattern Occurrence Detection

Detection of design pattern occurrences is part of several solutions to software engineering problems, and high accuracy of detection is important to help solve the actual problems. The improvement in accuracy of design pattern occurrence detection requires some way of evaluating various approaches. Currently, there are several different methods used in the community to evaluate accuracy. We show that these differences may greatly influence the accuracy results, which makes it nearly impossible to compare the quality of different techniques. We propose a benchmark suite to improve the situation and a community effort to contribute to, and evolve, the benchmark suite. Also, we propose fine-grained metrics assessing the accuracy of various approaches in the benchmark suite. This allows comparing the detection techniques and helps improve the accuracy of detecting design pattern occurrences.

[1]  Susan Elliott Sim,et al.  Using benchmarking to advance research: a challenge to software engineering , 2003, 25th International Conference on Software Engineering, 2003. Proceedings..

[2]  Sabine Buchholz,et al.  CoNLL-X Shared Task on Multilingual Dependency Parsing , 2006, CoNLL.

[3]  Michele Risi,et al.  Case studies of visual language based design patterns recovery , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[4]  Per Ahlgren,et al.  Measuring retrieval effectiveness with incomplete relevance data , 2006 .

[5]  Kyle G. Brown,et al.  Design reverse-engineering and automated design-pattern detection in Smalltalk , 1996 .

[6]  Niklas Pettersson.  Towards Pattern Detection in Software , 2006 .

[7]  Rudolf K. Keller,et al.  Pattern-based reverse-engineering of design components , 1999, Proceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002).

[8]  Jing Dong,et al.  Architecture and Design Pattern Discovery Techniques - A Review , 2007, Software Engineering Research and Practice.

[9]  Welf Löwe,et al.  Automatic design pattern detection , 2003, 11th IEEE International Workshop on Program Comprehension, 2003..

[10]  Ralph Johnson,et al.  design patterns elements of reusable object oriented software , 2019 .

[11]  Jörg Niere,et al.  Handling large search space in pattern-based reverse engineering , 2003, 11th IEEE International Workshop on Program Comprehension, 2003..

[12]  Rudolf Ferenc,et al.  Mining design patterns from C++ source code , 2003, International Conference on Software Maintenance, 2003. ICSM 2003. Proceedings..

[13]  Jim Welsh,et al.  Towards pattern-based design recovery , 2002, ICSE '02.

[14]  Dirk Beyer,et al.  Simple and efficient relational querying of software structures , 2003, 10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings..

[15]  Yann-Gaël Guéhéneuc,et al.  Recovering binary class relationships: putting icing on the UML cake , 2004, OOPSLA.

[16]  Matthias Meyer,et al.  Pattern-based Reengineering of Software Systems , 2006, 2006 13th Working Conference on Reverse Engineering.

[17]  Karel Driesen,et al.  The direct cost of virtual function calls in C++ , 1996, OOPSLA '96.

[18]  Tibor Gyimóthy,et al.  Towards a Benchmark for Evaluating Design Pattern Miner Tools , 2008, 2008 12th European Conference on Software Maintenance and Reengineering.

[19]  Giuliano Antoniol,et al.  Design pattern recovery in object-oriented software , 1998, Proceedings. 6th International Workshop on Program Comprehension. IWPC'98 (Cat. No.98TB100242).

[20]  Per Ahlgren,et al.  Evaluation of retrieval effectiveness with incomplete relevance data: Theoretical and experimental comparison of three measures , 2008, Inf. Process. Manag..

[21]  Lutz Prechelt,et al.  Design recovery by automated search for structural design patterns in object-oriented software , 1996, Proceedings of WCRE '96: 4rd Working Conference on Reverse Engineering.

[22]  Jing Dong,et al.  Verifying Behavioral Correctness of Design Pattern Implementation , 2008, SEKE.

[23]  Yann-Gaël Guéhéneuc,et al.  Efficient identification of design patterns with bit-vector algorithm , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[24]  Niklas Pettersson,et al.  Measuring precision for static and dynamic design pattern recognition as a function of coverage , 2005 .

[25]  Ralph Grishman,et al.  Evaluating Parsing Strategies Using Standardized Parse Files , 1992, ANLP.

[26]  Alessandro Orso,et al.  Recognizing behavioral patterns atruntime using finite automata , 2006, WODA '06.

[27]  Michele Risi,et al.  Design pattern recovery by visual language parsing , 2005, Ninth European Conference on Software Maintenance and Reengineering.

[28]  Ilka Philippow,et al.  An approach for reverse engineering of design patterns , 2004, Software & Systems Modeling.

[29]  Ronald A. Olsson,et al.  Reverse Engineering of Design Patterns from Java Source Code , 2006, 21st IEEE/ACM International Conference on Automated Software Engineering (ASE'06).

[30]  Ellen M. Voorhees,et al.  The Philosophy of Information Retrieval Evaluation , 2001, CLEF.

[31]  Treebanks.  Treebanks Building and Using Parsed Corpora , 2011 .

[32]  Joakim Nivre,et al.  On Evaluation of Accuracy in Pattern Detection , 2006 .

[33]  Yann-Gaël Guéhéneuc,et al.  A comparative framework for design recovery tools , 2006, Conference on Software Maintenance and Reengineering (CSMR'06).

[34]  Yongtao Sun,et al.  Design pattern detection by template matching , 2008, SAC '08.

[35]  Alexander Chatzigeorgiou,et al.  Design Pattern Detection Using Similarity Scoring , 2006, IEEE Transactions on Software Engineering.

[36]  Ulrik Pagh Schultz,et al.  Specialization patterns , 2000, Proceedings ASE 2000. Fifteenth IEEE International Conference on Automated Software Engineering.

[37]  Rudolf Ferenc,et al.  Evaluating C++ Design Pattern Miner Tools , 2006, 2006 Sixth IEEE International Workshop on Source Code Analysis and Manipulation.

[38]  Marek Vokác.  An efficient tool for recovering Design Patterns from C++ Code , 2006, J. Object Technol..

[39]  Ralph Grishman,et al.  A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars , 1991, HLT.

[40]  P. David Stotts,et al.  SPQR: flexible automated design pattern extraction from source code , 2003, 18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings..

[41]  Lothar Wendehals.  Improving Design Pattern Instance Recognition by Dynamic Analysis , 2003 .

[42]  Yann-Gaël Guéhéneuc,et al.  DeMIMA: A Multilayered Approach for Design Pattern Identification , 2008, IEEE Transactions on Software Engineering.

[43]  Andreas Elsner,et al.  Selective Tracing of Java Programs , 2004 .

[44]  Beatrice Santorini,et al.  Building a Large Annotated Corpus of English: The Penn Treebank , 1993, CL.

[45]  Welf Löwe,et al.  A Non-conservative Approach to Software Pattern Detection , 2007, 15th IEEE International Conference on Program Comprehension (ICPC '07).