Combining Estimates with Planning Poker--An Empirical Study

Combination of expert opinion is frequently used to produce estimates in software projects. However, if, when and how to combine expert estimates, is poorly understood. In order to study the effects of a combination technique called planning poker, the technique was introduced in a software project for half of the tasks. The tasks estimated with planning poker provided: 1) group consensus estimates that were less optimistic than the mechanical combination of individual estimates for the same tasks, and 2) group consensus estimates that were more accurate than the mechanical combination of individual estimates for the same tasks. The set of control tasks in the same project, estimated by individual experts, achieved similar estimation accuracy as the planning poker tasks. However, for both planning poker and the control group, measures of the median estimation bias indicated that both groups had unbiased estimates, as the typical estimated task was perfectly on target.

[1]  Magne Jørgensen,et al.  A framework for the analysis of software cost estimation accuracy , 2006, ISESE '06.

[2]  N. Nakagawa,et al.  Method to estimate parameter values in software prediction models , 1991 .

[3]  George Wright,et al.  Expert Opinions in Forecasting: The Role of the Delphi Technique , 2001 .

[4]  Urvashi Rathod,et al.  Defining ‘success’ for software projects: An exploratory revelation , 2006 .

[5]  Barry W. Boehm,et al.  Software Engineering Economics , 1993, IEEE Transactions on Software Engineering.

[6]  Mike Cohn,et al.  Agile Estimating and Planning , 2005 .

[7]  R. Buehler,et al.  Collaborative planning and prediction: Does group discussion affect optimistic biases in time estimation? , 2005 .

[8]  Stephen A. McGuire,et al.  Introductory Statistics , 2007, Technometrics.

[9]  S. Fiske,et al.  Social Psychology , 2019, Encyclopedia of Personality and Individual Differences.

[10]  Paul Strooper.  Proceedings of the 2005 Australian Software Engineering Conference , 2004 .

[11]  Timothy M. Franz,et al.  Management of Poor Performance: A Comparison of Manager, Group Member, and Group Disciplinary Decisions , 1999 .

[12]  Claes Wohlin,et al.  An experimental study of individual subjective effort estimations and combinations of the estimates , 1998, Proceedings of the 20th International Conference on Software Engineering.

[13]  Dag I. K. Sjøberg,et al.  The impact of customer expectation on software development effort estimates , 2004 .

[14]  Nils Christian Haugen.  An empirical study of using planning poker for user story estimation , 2006, AGILE 2006 (AGILE'06).

[15]  Robert T. Hughes,et al.  Expert judgement as an estimating method , 1996, Inf. Softw. Technol..

[16]  Barbara Kitchenham,et al.  Software Metrics: Measurement for Software Process Improvement , 1996 .

[17]  Richard E. Fairley.  Making Accurate Estimates , 2002, IEEE Softw..

[18]  Martin J. Shepperd,et al.  An experiment on software project size and effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[19]  G. W. Fischer.  When oracles fail—A comparison of four procedures for aggregating subjective probability forecasts , 1981 .

[20]  C. Powell.  The Delphi technique: myths and realities. , 2003, Journal of advanced nursing.

[21]  J. Scott Armstrong,et al.  How to Make Better Forecasts and Decisions: Avoid Face-to-Face Meetings , 2007 .

[22]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[23]  Watts S. Humphrey,et al.  Managing the software process , 1989, The SEI series in software engineering.

[24]  George Wright,et al.  The Delphi technique as a forecasting tool: issues and analysis , 1999 .

[25]  Johannes A. Zuber,et al.  Choice shift and group polarization : an analysis of the status of arguments and social decision schemes , 1992 .

[26]  R. L. Atkinson,et al.  Hilgard's Introduction to Psychology , 1999 .

[27]  Cemal Yilmaz,et al.  Software Metrics , 2008, Wiley Encyclopedia of Computer Science and Engineering.

[28]  Richard C. Atkinson,et al.  Hilgard's introduction to psychology, 12th ed. , 1996 .

[29]  Magne Jørgensen,et al.  Group Processes in Software Effort Estimation , 2004, Empirical Software Engineering.