Methodological insights from a rigorous small scale design experiment

This paper discusses the methods used to conduct high quality small-scale design experiments. It aims to provide a demonstrator promoting the uptake of more rigorous methods in design research and based on this it aims to specify a body of further work for linking study types and contexts. A small-scale experiment was conducted using methods specifically developed to mitigate four core problem areas identified from review: context, system understanding, methods and controls. The techniques were then critiqued in detail and used to draw several insights for design researchers including the value of control techniques and triangulation of metrics. Finally, the critique is used to specify further research aimed at linking design experimentation and design practice more effectively for design research.

[1]  Robert O. Briggs,et al.  On theory-driven design and deployment of collaboration systems , 2006, Int. J. Hum. Comput. Stud..

[2]  Chris Rogers,et al.  The benefits of model building in teaching engineering design , 2010 .

[3]  Suzanne T Bell,et al.  Deep-level composition variables as predictors of team performance: a meta-analysis. , 2007, The Journal of applied psychology.

[4]  C. D. De Dreu,et al.  Task versus relationship conflict, team performance, and team member satisfaction: a meta-analysis. , 2003, The Journal of applied psychology.

[5]  Stephen Culley,et al.  The Challenges Facing Ethnographic Design Research: A Proposed Methodological Solution , 2009 .

[6]  Barbara Senior.  Team roles and team performance: Is there ‘really’ a link? , 1997 .

[7]  Hsin-Ginn Hwang,et al.  The effect of group size on group performance in computer-supported decision making , 1994, Inf. Manag..

[8]  J. Lave.  Cognition in Practice: Outdoors: a social anthropology of cognition in practice , 1988 .

[9]  Chris W. Clegg,et al.  Design engineering competencies: future requirements and predicted changes in the forthcoming decade , 2005 .

[10]  Mark Robinson.  Work sampling: Methodological advances and new applications , 2010 .

[11]  Maaike Kleinsmann,et al.  Performing high quality research into design practice , 2009 .

[12]  Nigel Cross,et al.  Forty years of design research , 2007 .

[13]  Wayne D. Gray,et al.  Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods , 1998, Hum. Comput. Interact..

[14]  B. Baltes,et al.  Computer-Mediated Communication and Group Decision Making: A Meta-Analysis , 2002 .

[15]  Ben J Hicks,et al.  Designer behaviour and activity: an industrial observation method , 2011 .

[16]  Sallie M. Henry,et al.  Using Belbin's leadership role to improve team effectiveness: An empirical investigation , 1999, J. Syst. Softw..

[17]  Thomas J. Howard,et al.  Reuse of ideas and concepts for creative stimuli in engineering design , 2011 .

[18]  Richard A. Guzzo,et al.  Teams in organizations: recent research on performance and effectiveness. , 1996, Annual review of psychology.

[19]  Nigel Cross,et al.  Design cognition: results from protocol and other empirical studies of design activity , 2016 .

[20]  D. J. Magin,et al.  Peer tutoring in engineering design: A case study , 1995 .

[21]  A. Osborn.  Applied imagination : principles and procedures of creative problem-solving , 1957 .

[22]  B. Bender,et al.  Application of Methods from Social Sciences in Design Research , 2002 .

[23]  Belinda López-Mesa,et al.  Effects of additional stimuli on idea-finding in design teams , 2011 .

[24]  Tore Dybå,et al.  Empirical studies of agile software development: A systematic review , 2008, Inf. Softw. Technol..

[25]  Tom Cassidy,et al.  Seeing and discovering: how do student designers reinterpret sketches and digital marks during graphic design ideation? , 2010 .

[26]  Helen Sharp,et al.  Collaboration and co-ordination in mature eXtreme programming teams , 2008, Int. J. Hum. Comput. Stud..

[27]  Robert O. Briggs,et al.  On The Relationship Between Idea-Quantity and Idea-Quality During Ideation , 2008 .

[28]  George P. Huber,et al.  Effects of size and spatial arrangements on group decision making. , 1974 .

[29]  Anit Somech,et al.  Understanding team innovation: The role of team processes and structures. , 2001 .

[30]  Filippo Lanubile,et al.  Empirical Evaluation of Software Maintenance Technologies , 1997, Empirical Software Engineering.

[31]  Lucienne Blessing,et al.  DRM, a Design Research Methodology , 2009 .

[32]  Shari Lawrence Pfleeger,et al.  Preliminary Guidelines for Empirical Research in Software Engineering , 2002, IEEE Trans. Software Eng..

[33]  Ken Friedman,et al.  Theory construction in design research: criteria: approaches, and methods , 2003 .

[34]  Poul Henrik Kyvsgaard Hansen,et al.  The Product Development Process Ontology: Creating a Learning Research Community , 2001 .

[35]  Roderick M. Kramer,et al.  Choice behavior in social dilemmas: Effects of social identity, group size, and decision framing. , 1986 .

[36]  Françoise Détienne,et al.  Quantitative Measurements of the Influence of Participant Roles during Peer Review Meetings , 2001, Empirical Software Engineering.

[37]  Petra Badke-Schaub,et al.  Thinking in design teams - an analysis of team communication , 2002 .

[38]  Edward Elias,et al.  Designing for ‘use phase’ energy losses of domestic products , 2009 .

[39]  J. G. Adair,et al.  The Placebo Control Group: An Analysis of Its Effectiveness in Educational Research , 1990 .

[40]  John S. Gero,et al.  The differences between retrospective and concurrent protocols in revealing the process-oriented aspects of the design process , 2001 .

[41]  G. Goldschmidt,et al.  How good are good ideas? Correlates of design creativity , 2005 .

[42]  S. Iliffe,et al.  Bmc Medical Research Methodology Open Access the Hawthorne Effect: a Randomised, Controlled Trial , 2007 .

[43]  Deborah G. . Ancona,et al.  Demography and Design: Predictors of New Product Team Performance , 1992 .

[44]  K. Emmons,et al.  How can we increase translation of research into practice? Types of evidence needed. , 2007, Annual review of public health.

[45]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[46]  Narasimhaiah Gorla,et al.  Who should work with whom?: building effective software project teams , 2004, CACM.

[47]  D. Torgerson,et al.  Avoiding Bias in Randomised Controlled Trials in Educational Research , 2003 .

[48]  Nigel Cross,et al.  Analysing design activity , 1996 .

[49]  L. Ball.  Applying ethnography in the analysis and support of expertise in engineering design , 2000 .

[50]  Gina J. Medsker,et al.  RELATIONS BETWEEN WORK GROUP CHARACTERISTICS AND EFFECTIVENESS: IMPLICATIONS FOR DESIGNING EFFECTIVE WORK GROUPS , 1993 .

[51]  Leonard Adelman,et al.  Experiments, quasi-experiments, and case studies: A review of empirical methods for evaluating decision support systems , 1991, IEEE Trans. Syst. Man Cybern..

[52]  Matthew I. Campbell,et al.  An experimental study on the effects of a computational design tool on concept generation , 2009 .

[53]  Paul Leber,et al.  The use of placebo control groups in the assessment of psychiatric drugs: an historical context , 2000, Biological Psychiatry.

[54]  Eduardo Salas,et al.  Does Team Training Improve Team Performance? A Meta-Analysis , 2008, Hum. Factors.

[55]  Patrick Dillon,et al.  Creativity, Integrativism and a Pedagogy of Connection. , 2006 .

[56]  Nancy L. Leech,et al.  Linking Research Questions to Mixed Methods Data Analysis Procedures 1 , 2006 .

[57]  Shinobu Yoshimura,et al.  11th International Conference on Engineering Design , 1997 .

[58]  Helen Sharp,et al.  Ethnographically-informed empirical studies of software practice , 2007, Inf. Softw. Technol..

[59]  L. R. Hoffman.  Group Problem Solving1 , 1965 .

[60]  J. Hackman,et al.  Effects of size and task type on group performance and member reactions , 1970 .

[61]  Murray R. Barrick,et al.  Relating member ability and personality to work-team processes and team effectiveness. , 1998 .

[62]  Michael D. Myers,et al.  A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems , 1999, MIS Q..

[63]  G. Stewart.  A Meta-Analytic Review of Relationships Between Team Design Features and Team Performance , 2006 .

[64]  Jan A.M. Corremans,et al.  Measuring the effectiveness of a design method to generate form alternatives: an experiment performed with freshmen students product development , 2011 .

[65]  Thomas Cook,et al.  Where does good evidence come from? , 2007 .

[66]  W. Newstetter,et al.  Design Knowing and Learning: Cognition in Design Education , 2001 .

[67]  Daniel Collado-Ruiz,et al.  Influence of environmental information on creativity , 2010 .

[68]  Andy Dong,et al.  The latent semantic approach to studying design team communication , 2005 .

[69]  A. Hare.  A Study of Interaction and Consensus in Different Sized Groups , 1952 .

[70]  Ellen Yi-Luen Do,et al.  Extended linkography and distance graph in design evaluation: an empirical study of the dual effects of inspiration sources in creative design , 2010 .