Characterising the relationship between practice and laboratory-based studies of designers for critical design situations

Experimental study of the designer plays a critical role in design research. However laboratory based study is often poorly compared and contrasted to practice, leading to a lack of uptake and subsequent research impact. The importance of addressing this issue is highlighted by its significant influence on design research and many related fields. As such the main aim of this work is to improve empirical design research by characterising the relationship between practice and laboratory-based studies for critical design situations. A review of the state of the art methods in design research and key related fields is reported. This highlights the importance and commonality of a set or core issues connected to the failure to effectively link study of practice and study in the laboratory. Further to this a technical review and scoping was carried out to establish the most efective capture strategy to be used when studying the designer empirically. Subsequently three studies are reported, forming a three point comparison between practice the laboratory (with student practitioners) and an intermediary case (a laboratory with practitioners) . Results from these studies contextualise the critical situations in practice and develop a detailed multi-level comparison between practice and the laboratory which was then validated with respect to a number of existing studies. The primary contribution of this thesis is the development of a detailed multi-level relationship between practice and the laboratory for critical design situations: information seeking, ideation and design review. The second key contribution is the development of a generic method for the empirical study of designers in varying contexts - allowing researchers to build on this work and more effectively link diverse studies together. The final key contribution of this work is the identification of a number of core methodological issues and mitigating techniques affecting both design research and its related fields.

[1]  J. Frank,et al.  Psychotherapy and the placebo effect. , 2011, Psychological bulletin.

[2]  Hamish McAlpine,et al.  Improving the management of Informal Engineering Information through Electronic Logbooks , 2010 .

[3]  Ben J Hicks,et al.  An investigation into the use and content of the engineer's logbook , 2006 .

[4]  Janice Singer,et al.  Studying Software Engineers: Data Collection Techniques for Software Field Studies , 2005, Empirical Software Engineering.

[5]  Timothy P. McDonald,et al.  A demonstration of an advanced library based approach to the initial design exploration of different hullform configurations , 2012, Comput. Aided Des..

[6]  Francesca Montagna,et al.  A hybrid-approach to model design and development processes and the evaluation of innovation opportunities , 2008, Int. J. Manuf. Technol. Manag..

[7]  Fumihiko Kimura,et al.  Design methodologies: Industrial and educational applications , 2009 .

[8]  S. Reed.  Structure and Behaviour: Extending Duverger's Law to the Japanese Case , 1990, British Journal of Political Science.

[9]  Rachael Luck,et al.  Learning to talk to users in participatory design situations , 2007 .

[10]  Derek F. Sheldon.  Design Review 2005/2006—The ever increasing maturity of design research papers and case studies , 2006 .

[11]  Gavriel Salomon,et al.  Transcending the Qualitative-Quantitative Debate: The Analytic and Systemic Approaches to Educational Research , 1991 .

[12]  G. Goodwin,et al.  Does Team Training Improve Team Performance , 2008 .

[13]  Matthew B. Miles,et al.  Qualitative Data Analysis: An Expanded Sourcebook , 1994 .

[14]  Peter Bryant,et al.  Design Experiments and Laboratory Approaches to Learning: Steps Toward Collaborative Exchange , 2003 .

[15]  Janet McDonnell,et al.  Analysing Design Behaviour: The Design Thinking Research Symposia Series , 2007 .

[16]  D. Torgerson,et al.  Avoiding Bias in Randomised Controlled Trials in Educational Research , 2003 .

[17]  Mark Robinson.  The Development and Piloting of a Novel Methodology for Determining How Designers Spend Their Working Time , 2005 .

[18]  Derek F. Sheldon.  A review on the relevance of design science in a global product development arena , 2004 .

[19]  William R. Shadish,et al.  Propensity Scores and Quasi-Experiments: A Testimony to the Practical Side of Lee Sechrest. , 2006 .

[20]  D. C. Phillips,et al.  On the Science of Education Design Studies , 2003 .

[21]  E. Mayo.  The Human Problems of an Industrial Civilization , 1934, Nature.

[22]  P. Slater.  Contrasting Correlates of Group Size , 1958 .

[23]  Owain Pedgley,et al.  Capturing and analysing own design activity , 2007 .

[24]  J. M. Olson,et al.  An expectancy-attribution model of the effects of placebos. , 1981, Psychological review.

[25]  Ben J Hicks,et al.  Comparing formal and informal documents: lessons for more complete design records , 2009 .

[26]  Toon W. Taris,et al.  Bricks without clay: On urban myths in occupational health psychology , 2006 .

[27]  Mattias Lindahl,et al.  Engineering designers’ experience of design for environment methods and tools : Requirement definitions from an interview study , 2006 .

[28]  Belinda López-Mesa,et al.  Effects of additional stimuli on idea-finding in design teams , 2011 .

[29]  Mark Robinson.  Work sampling: Methodological advances and new applications , 2010 .

[30]  Janice Singer,et al.  An examination of software engineering work practices , 1997, CASCON.

[31]  anonymous,et al.  Comprehensive review , 2019 .

[32]  Judith S. Olson,et al.  Small Group Design Meetings: An Analysis of Collaboration , 1992, Hum. Comput. Interact..

[33]  Amrik S. Sohal,et al.  Generative and case study research in quality management: part I: theoretical considerations , 1996 .

[34]  Yoram Reich,et al.  THE N-DIM APPROACH TO CREATING DESIGN SUPPORT SYSTEMS , 1997 .

[35]  Industrial Strategy.  Business population estimates for the UK and regions , 2011 .

[36]  A. Pettigrew.  Longitudinal Field Research on Change: Theory and Practice , 1990 .

[37]  Robert P. Smith,et al.  An Observational Study of Design Team Process: A Comparison of Student and Professional Engineers , 1998 .

[38]  S. Iliffe,et al.  Bmc Medical Research Methodology Open Access the Hawthorne Effect: a Randomised, Controlled Trial , 2022 .

[39]  J. Lynch.  Theory and external validity , 1999 .

[40]  Steven M. Smith,et al.  Metrics for measuring ideation effectiveness , 2003 .

[41]  Philip E. T. Lewis,et al.  Research Methods for Business Students , 2006 .

[42]  Carolyn Riehl,et al.  Feeling Better: A Comparison of Medical Research and Education Research , 2006 .

[43]  Alistair Lowe.  Studies of information use by engineering designers and the development of strategies to aid in its classification and retrieval , 2002 .

[44]  M. Miles,et al.  Drawing Valid Meaning from Qualitative Data: Toward a Shared Craft , 1984 .

[45]  J. Knottnerus,et al.  Real world research. , 2010, Journal of clinical epidemiology.

[46]  D. Silverman,et al.  Ensuring rigour in qualitative research , 1997 .

[47]  Peter Lynch,et al.  Adolescent smoking—an alternative perspective using personal construct theory , 1995 .

[48]  D. Nathan.  Clinical Research: Perceptions, Reality, and Proposed Solutions , 1998 .

[49]  Reinhard Schütte,et al.  The Guidelines of Modeling - An Approach to Enhance the Quality in Information Models , 1998, ER.

[50]  Amaresh Chakrabarti,et al.  Towards an ‘ideal’ approach for concept generation , 2003 .

[51]  Jim Gemmell,et al.  Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences , 2004, MM 2004.

[52]  K. Merrell.  Better Methods, Better Solutions: Developments in School-Based Behavioral Assessment , 2010 .

[53]  Ritchie Macefield,et al.  Usability studies and the Hawthorne effect , 2007 .

[54]  Mark A. Robinson,et al.  An empirical analysis of engineers' information behaviors , 2010, J. Assoc. Inf. Sci. Technol..

[55]  John A. List,et al.  What Do Laboratory Experiments Tell Us About the Real World , 2006 .

[56]  S. Pużyński.  Placebo in the investigation of psychotropic drugs, especially antidepressants , 2004, Science and engineering ethics.

[57]  P. F. Secord,et al.  Implications for psychology of the new philosophy of science. , 1983 .

[58]  John B. Reid,et al.  Effects of Instructional Set and Experimenter Influence on Observer Reliability. , 1973 .

[59]  Abigail Sellen,et al.  How knowledge workers use the web , 2002, CHI.

[60]  Steve Culley,et al.  An analysis of the content of technical information used by engineering designers , 2000 .

[61]  Dominique Scaravetti,et al.  Design space exploration in embodiment design: an application to the design of aircraft air conditioners , 2009 .

[62]  C. Shalley,et al.  WHAT LEADERS NEED TO KNOW: A REVIEW OF SOCIAL AND CONTEXTUAL FACTORS THAT CAN FOSTER OR HINDER CREATIVITY , 2004 .

[63]  Pirita Seitamaa-Hakkarainen,et al.  Composition and construction in experts' and novices' weaving design , 2001 .

[64]  David Maffin,et al.  Engineering Design Models: Context, theory and practice , 1998 .

[65]  Peter Törlind,et al.  Towards true collaboration in global design teams , 2005 .

[66]  Wayne B Jonas,et al.  Deconstructing the Placebo Effect and Finding the Meaning Response , 2002, Annals of Internal Medicine.

[67]  Ac Rianne Valkenburg,et al.  The reflective practice of design teams , 1998 .

[68]  P. Lloyd,et al.  Can concurrent verbalization reveal design cognition , 1995 .

[69]  Maria Franca Norese,et al.  Identification and Development of Alternatives: Introduction to the Recognition of Process Typologies , 1989 .

[70]  Françoise Détienne,et al.  Analysing Viewpoints in Design through the Argumentation Process , 2001, INTERACT.

[71]  K. Malterud.  Qualitative research: standards, challenges, and guidelines , 2001, The Lancet.

[72]  Bernhard Schmitz,et al.  New perspectives for the evaluation of training sessions in self-regulated learning: Time-series analyses of diary data , 2006 .

[73]  Trudy van der Weijden,et al.  Block design allowed for control of the Hawthorne effect in a randomized controlled trial of test ordering. , 2004, Journal of clinical epidemiology.

[74]  D. Stryer,et al.  Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. , 2003, JAMA.

[75]  Alex H. B. Duffy,et al.  Supporting `Design for Re-use' with Modular Design , 2007, Concurr. Eng. Res. Appl..

[76]  Chris W. Clegg,et al.  Design engineering competencies: future requirements and predicted changes in the forthcoming decade , 2005 .

[77]  Bryan Lawson,et al.  About: Designing – Analysing Design Meetings , 2010 .

[78]  W. Shadish,et al.  Experimental and Quasi-Experimental Designs for Generalized Causal Inference , 2001 .

[79]  Peter Jeppsson,et al.  Collaboration environments for distributed engineering : development of a prototype system , 1999 .

[80]  C. McMahon,et al.  The Multiple Traditions of Design Research , 2011 .

[81]  William Lidwell,et al.  Universal principles of design : 100 ways to enhance usability,influence perception, increase appeal, make better, designdecisions, and teach through design , 2003 .

[82]  Ben J Hicks,et al.  The digital divide: Investigating the personal information management practices of engineers , 2011 .

[83]  Mattias Bergström,et al.  Lessons learned and future challenges for design observatory research , 2009 .

[84]  Remko van der Lugt.  Functions of sketching in design idea generation meetings , 2002, Creativity & Cognition.

[85]  Joshua D. Summers,et al.  An experimental methodology for investigating communication in collaborative design review meetings , 2005 .

[86]  Gary Doolittle,et al.  Telehealth services to improve nonadherence: A placebo-controlled study. , 2006, Telemedicine journal and e-health : the official journal of the American Telemedicine Association.

[87]  R. Horwitz,et al.  Placebos, active control groups, and the unpredictability paradox , 2000, Biological Psychiatry.

[88]  B. Zimmerman.  Investigating Self-Regulation and Motivation: Historical Background, Methodological Developments, and Future Prospects , 2008, American Educational Research Journal.

[89]  P. Suedfeld.  The subtractive expectancy placebo procedure: a measure of non-specific factors in behavioural interventions. , 1984, Behaviour research and therapy.

[90]  Terry Winograd,et al.  THE iLOFT PROJECT: A TECHNOLOGICALLY ADVANCED COLLABORATIVE DESIGN WORKSPACE AS RESEARCH INSTRUMENT , 2003 .

[91]  Duncan Sanderson,et al.  Coordinating joint design work: the role of communication and artefacts , 1998 .

[92]  Ben J Hicks,et al.  Investigating the structure and organisation of engineering logbooks for more effective re-use , 2008 .

[93]  Helen Sharp,et al.  Collaboration and co-ordination in mature eXtreme programming teams , 2008, Int. J. Hum. Comput. Stud..

[94]  Denis Newman,et al.  Can Scientific Research From the Laboratory be of Any Use to Teachers? , 2004 .

[95]  Robert O. Briggs,et al.  On The Relationship Between Idea-Quantity and Idea-Quality During Ideation , 2008 .

[96]  Ingrid Mulder,et al.  Assessing group learning and shared understanding in technology-mediated interaction , 2002, J. Educ. Technol. Soc..

[97]  Peter Törlind,et al.  Re-experiencing engineering meetings : knowledge reuse challenges from virtual meetings , 2006 .

[98]  Dave Stewart,et al.  Waypoint: An Integrated Search and Retrieval System for Engineering Documents , 2004, J. Comput. Inf. Sci. Eng..

[99]  Andrés Catena,et al.  Criteria of the peer review process for publication of experimental and quasi-experimental research in Psychology: A guide for creating research papers , 2006 .

[100]  N. Roll-Hansen.  Studying natural science without nature? Reflections on the realism of so-called laboratory studies , 1998 .

[101]  P. Renne,et al.  2.5-million-year-old stone tools from Gona, Ethiopia , 1997, Nature.

[102]  T. Kaptchuk,et al.  The power of context: reconceptualizing the placebo effect , 2008, Journal of the Royal Society of Medicine.

[103]  Mcdp Mathieu Weggeman,et al.  Towards an innovation culture : what are it's national, corporate, marketing and engineering aspects, some experimental evidence , 2001 .

[104]  Nancy L. Leech,et al.  Linking Research Questions to Mixed Methods Data Analysis Procedures 1 , 2006 .

[105]  Laurent Gerbaud,et al.  An observational study of multi-disciplinary co-design – application to an electromechanical device , 2007 .

[106]  Larry Leifer,et al.  210-NP: measuring the mechanical engineering design process , 1996, Technology-Based Re-Engineering Engineering Education Proceedings of Frontiers in Education FIE'96 26th Annual Conference.

[107]  Helen Sharp,et al.  Ethnographically-informed empirical studies of software practice , 2007, Inf. Softw. Technol..