Time pressure: a controlled experiment of test case development and requirements review

Time pressure is prevalent in the software industry in which shorter and shorter deadlines and high customer demands lead to increasingly tight deadlines. However, the effects of time pressure have received little attention in software engineering research. We performed a controlled experiment on time pressure with 97 observations from 54 subjects. Using a two-by-two crossover design, our subjects performed requirements review and test case development tasks. We found statistically significant evidence that time pressure increases efficiency in test case development (high effect size Cohen’s d=1.279) and in requirements review (medium effect size Cohen’s d=0.650). However, we found no statistically significant evidence that time pressure would decrease effectiveness or cause adverse effects on motivation, frustration or perceived performance. We also investigated the role of knowledge but found no evidence of the mediating role of knowledge in time pressure as suggested by prior work, possibly due to our subjects. We conclude that applying moderate time pressure for limited periods could be used to increase efficiency in software engineering tasks that are well structured and straight forward.

[1]  R. Yerkes,et al.  The relation of strength of stimulus to rapidity of habit‐formation , 1908 .

[2]  Richard Fisher,et al.  Students as surrogates for practicing accountants: Further evidence , 2012 .

[3]  Brian C. Spilker,et al.  The effects of time pressure and knowledge on key word selection behavior in tax research , 1995 .

[4]  Claes Wohlin,et al.  Using students as subjects - an empirical evaluation , 2008, ESEM '08.

[5]  Per Runeson,et al.  Using Students as Experiment Subjects – An Analysis on Graduate and Freshmen Student Data , 2003 .

[6]  Saurabh Sinha,et al.  Global software testing under deadline pressure: Vendor-side experiences , 2014, Inf. Softw. Technol..

[7]  Joseph G. Johnson,et al.  Take The First: Option-generation and resulting choices , 2003 .

[8]  Barry L. Lewis,et al.  Determinants of Auditor Expertise , 1990 .

[9]  Westley Weimer,et al.  A human study of fault localization accuracy , 2010, 2010 IEEE International Conference on Software Maintenance.

[10]  I. Steiner.  Group process and productivity , 1972 .

[11]  Kjetil Moløkken-Østvold,et al.  A review of software surveys on software effort estimation , 2003, 2003 International Symposium on Empirical Software Engineering, 2003. ISESE 2003. Proceedings..

[12]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[13]  Walter F. Tichy,et al.  Hints for Reviewing Empirical Work in Software Engineering , 2000, Empirical Software Engineering.

[14]  Mika Mäntylä,et al.  More testers - The effect of crowd size and time restriction in software testing , 2013, Inf. Softw. Technol..

[15]  Ning Nan,et al.  Impact of Budget and Schedule Pressure on Software Development Cycle Time and Effort , 2009, IEEE Transactions on Software Engineering.

[16]  Steven M. Glover.  The influence of time pressure and accountability on auditors' processing of nondiagnostic information , 1997 .

[17]  Tracy Hall,et al.  Perceptions of software quality: a pilot study , 1998, Software Quality Journal.

[18]  C. Dreu.  Time pressure and closing of the mind in negotiation , 2003 .

[19]  Tracy Hall,et al.  De-motivators for software process improvement: an analysis of practitioners' views , 2003, J. Syst. Softw..

[20]  Marci S. DeCaro,et al.  From poor performance to success under stress: working memory, strategy selection, and mathematical problem solving under pressure. , 2007, Journal of experimental psychology. Learning, memory, and cognition.

[21]  Mark C. Paulk,et al.  The Impact of Design and Code Reviews on Software Quality: An Empirical Study Based on PSP Data , 2009, IEEE Transactions on Software Engineering.

[22]  Tore Dybå,et al.  A systematic review of statistical power in software engineering experiments , 2006, Inf. Softw. Technol..

[23]  Ching-I Teng,et al.  Interactive effects of nurse-experienced time pressure and burnout on patient safety: a cross-sectional survey. , 2010, International journal of nursing studies.

[24]  Kjetil Molkken,et al.  A Review of Surveys on Software Effort Estimation , 2003 .

[25]  Tony Gorschek,et al.  Test-case Driven versus Checklist-based Inspections of Software Requirements - An Experimental Evaluation , 2007, WER.

[26]  Linda S. McDaniel,et al.  The Effects Of Time Pressure And Audit Program Structure On Audit Performance , 1990 .

[27]  Magne Jørgensen,et al.  Impact of effort estimates on software project work , 2001, Inf. Softw. Technol..

[28]  H. Zur,et al.  The effect of time pressure on risky choice behavior , 1981 .

[29]  Matthias Sutter,et al.  Time is Money - Time Pressure, Incentives, and the Quality of Decision-Making , 2004 .

[30]  Stefan Biffl,et al.  Investigating the Defect Detection Effectiveness and Cost Benefit of Nominal Inspection Teams , 2003, IEEE Trans. Software Eng..

[31]  Gary Klein,et al.  Time Pressure, Skill, and Move Quality in Chess , 1988 .

[32]  Natalia Juristo Juzgado,et al.  Using differences among replications of software engineering experiments to gain knowledge , 2009, 2009 3rd International Symposium on Empirical Software Engineering and Measurement.

[33]  Claes Wohlin,et al.  Experimentation in software engineering: an introduction , 2000 .

[34]  Nevena T. Koukova,et al.  COPING WITH TIME PRESSURE AND KNOWLEDGE SHARING IN BUYER–SUPPLIER RELATIONSHIPS , 2011 .

[35]  Tore Dybå,et al.  Inferring Skill from Tests of Programming Performance: Combining Time and Quality , 2011, 2011 International Symposium on Empirical Software Engineering and Measurement.

[36]  Joseph S. Valacich,et al.  The effects of task complexity and time availability limitations on human performance in database query tasks , 2005, Int. J. Hum. Comput. Stud..

[37]  Amela Karahasanovic,et al.  A survey of controlled experiments in software engineering , 2005, IEEE Transactions on Software Engineering.

[38]  Jeffrey C. Carver,et al.  Perspective-Based Reading: A Replicated Experiment Focused on Individual Reviewer Effectiveness , 2006, Empirical Software Engineering.

[39]  Jeffrey C. Carver,et al.  The effect of the number of inspectors on the defect estimates produced by capture-recapture models , 2008, 2008 ACM/IEEE 30th International Conference on Software Engineering.

[40]  Patrik Berander,et al.  Using students as subjects in requirements prioritization , 2004, Proceedings. 2004 International Symposium on Empirical Software Engineering, 2004. ISESE '04..

[41]  J. Sweller.  COGNITIVE LOAD THEORY, LEARNING DIFFICULTY, AND INSTRUCTIONAL DESIGN , 1994 .

[42]  Foutse Khomh,et al.  On Rapid Releases and Software Testing , 2013, 2013 IEEE International Conference on Software Maintenance.

[43]  Eric S. Williams,et al.  Managed care, time pressure, and physician job satisfaction: Results from the physician worklife study , 2000, Journal of General Internal Medicine.

[44]  Helen Sharp,et al.  Motivation in Software Engineering: A systematic literature review , 2008, Inf. Softw. Technol..

[45]  Niklaus Wirth,et al.  A Plea for Lean Software , 1995, Computer.

[46]  Jacob Cohen.  Statistical Power Analysis for the Behavioral Sciences , 1969, The SAGE Encyclopedia of Research Design.

[47]  Felix C. Brodbeck,et al.  Stressor-burnout relationship in software development teams , 1994 .

[48]  SinhaSaurabh,et al.  Global software testing under deadline pressure , 2014 .

[49]  Robert D. Austin,et al.  The Effects of Time Pressure on Quality in Software Development: An Agency Model , 2001, Inf. Syst. Res..

[50]  Robert L. Glass,et al.  Project retrospectives, and why they never happen , 2002, IEEE Software.

[51]  N. Cliff.  Dominance statistics: Ordinal analyses to answer ordinal questions. , 1993 .

[52]  Nina Mazar,et al.  Large stakes and big mistakes , 2009 .

[53]  Aybüke Aurum,et al.  When Product Managers Gamble with Requirements: Attitudes to Value and Risk , 2009, REFSQ.

[54]  Scott H. Costello.  Software engineering under deadline pressure , 1984, SOEN.

[55]  Jeffrey C. Carver,et al.  Issues in using students in empirical studies in software engineering education , 2003, Proceedings. 5th International Workshop on Enterprise Networking and Computing in Healthcare Industry (IEEE Cat. No.03EX717).

[56]  Claes Wohlin,et al.  Using Students as Subjects—A Comparative Study of Students and Professionals in Lead-Time Impact Assessment , 2000, Empirical Software Engineering.

[57]  Gordon B. Davis,et al.  Can Humans Detect Errors in Data? Impact of Base Rates, Incentives, and Goals , 1997, MIS Q..

[58]  Michael Hoerger,et al.  When does haste make waste? Speed-accuracy tradeoff, skill level, and the tools of the trade. , 2008, Journal of experimental psychology. Applied.