Real-time feedback improves multi-stakeholder design for complex environmental systems

We test whether providing quantitative real-time feedback relating design decisions to system objectives improves group solutions in an interdependent energy-water design task. While prior research suggests an important role of real-time feedback on task performance, few studies have examined the role of real-time feedback in the design of complex environmental systems. We tested a real-time feedback approach using a mixed within- and between-subject experiment (n = 88 Carnegie Mellon University students, divided into 22 groups of four). When compared to individual designs and informal collaborations, real-time performance feedback yielded solutions closer to the Pareto frontier and reduced both financial (by 26% and 21%) and environmental cost (by 34% and 12%). In addition, informal collaboration did not improve group decision-making when compared to individual designs. The results suggest that optimal solutions to meeting energy and water demand while minimizing cost and environmental impact can be obscured in informal collaborations, but that real-time feedback to system designers can help avoid waste of public resources.

[1]  R. Addo-Tenkorang.  Concurrent Engineering (CE): A Review Literature Report , 2011 .

[2]  Sanford Weisberg,et al.  An R Companion to Applied Regression , 2010 .

[3]  Bahador Bahrami,et al.  Aggregated knowledge from a small number of debates outperforms the wisdom of large crowds , 2017, Nature Human Behaviour.

[4]  Mark A. Neerincx,et al.  Effects of different real-time feedback types on human performance in high-demanding work conditions , 2016, Int. J. Hum. Comput. Stud..

[5]  Jane M. Booker,et al.  Solving black box computation problems using expert knowledge theory and methods , 2004, Reliab. Eng. Syst. Saf..

[6]  R. Wheeler,et al.  The NASA Exploration Design Team: blueprint for a new design paradigm , 2005, 2005 IEEE Aerospace Conference.

[7]  Maria C. Yang,et al.  A Study of Student Design Team Behaviors in Complex System Design , 2012, Journal of Mechanical Design.

[8]  C. Frith,et al.  Together, Slowly but Surely , 2011, Journal of experimental psychology. Human perception and performance.

[9]  Timothy V. Bartholomew,et al.  Multiobjective Optimization Model for Minimizing Cost and Environmental Impact in Shale Gas Water and Wastewater Management , 2016 .

[10]  S. Courtright,et al.  Channeled Autonomy , 2016 .

[11]  Sally Andrews,et al.  To transform or not to transform: using generalized linear mixed models to analyse reaction time data , 2015, Front. Psychol..

[12]  Utpal Bose,et al.  Design and evaluation of a group support system supported process to resolve cognitive conflicts , 2015, Comput. Hum. Behav..

[13]  M. Ramachandran,et al.  Application of multi-criteria decision making to sustainable energy planning--A review , 2004 .

[14]  Achim Zeileis,et al.  Various versatile variances : An object-oriented implementation of clustered covariances in R Working , 2017 .

[15]  Joseph S. Valacich,et al.  Enhancing the Motivational Affordance of Information Systems: The Effects of Real-Time Performance Feedback and Goal Setting in Group Collaboration Environments , 2010, Manag. Sci..

[16]  D. Barr,et al.  Limits on theory of mind use in adults , 2003, Cognition.

[17]  D. Baker.  Enhancing Group Decision Making: An Exercise to Reduce Shared Information Bias , 2010 .

[18]  Holger R. Maier,et al.  Future research challenges for incorporation of uncertainty in environmental and ecological decision-making , 2008 .

[19]  Bernard A. Nijstad,et al.  Persistence of Brainstorming Groups: How Do People Know When to Stop? , 1999 .

[20]  Philip Leifeld,et al.  texreg: Conversion of Statistical Model Output in R to LaTeX and HTML Tables , 2013 .

[21]  Mary F. Johnson,et al.  Non‐inferiority trials: the ‘at least as good as’ criterion , 2002, Statistics in medicine.

[22]  Peter E. Latham,et al.  Does interaction matter? Testing whether a confidence heuristic can replace interaction in collective decision-making , 2014, Consciousness and Cognition.

[23]  Michael A. Hunter,et al.  Sampling and generalisability in developmental research: Comparison of random and convenience samples of older adults , 2002 .

[24]  M. Segers,et al.  The effect of team feedback and guided reflexivity on team performance change , 2014 .

[25]  Robert J. Barry,et al.  Individual differences in task-related activation and performance , 2009, Physiology & Behavior.

[26]  Jeffrey L. Smith.  Concurrent Engineering in the Jet Propulsion Laboratory Project Design Center , 1998 .

[27]  Maarten A. S. Boksem,et al.  Effects of mental fatigue on attention: an ERP study. , 2005, Brain research. Cognitive brain research.

[28]  C. Frith,et al.  What failure in collective decision-making tells us about metacognition , 2012, Philosophical Transactions of the Royal Society B: Biological Sciences.

[29]  K. Williams,et al.  Many Hands Make Light the Work: The Causes and Consequences of Social Loafing , 1979 .

[30]  Wim J. C. Verhagen,et al.  Concurrent Engineering in the 21st Century , 2015 .

[31]  Florian Ederer,et al.  Feedback and Motivation in Dynamic Tournaments , 2010 .

[32]  R. Hogarth,et al.  Learning from feedback: exactingness and incentives. , 1991, Journal of experimental psychology. Learning, memory, and cognition.

[33]  Annie Lang.  Involuntary Attention and Physiological Arousal Evoked by Structural Features and Emotional Content in TV Commercials , 1990 .

[34]  Pablo Bermell-Garcia,et al.  Knowledge-Based Engineering , 2015 .

[35]  E. Scott Geller,et al.  The Additive Impact of Group and Individual Publicly Displayed Feedback: Examining Individual Response Patterns and Response Generalization in a Safe-Driving Occupational Intervention , 2010, Behavior modification.

[36]  Peter Wolf,et al.  Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning , 2014, Experimental Brain Research.

[37]  Gloria Mark,et al.  Extreme collaboration , 2002, CACM.

[38]  R. Arritt,et al.  Integrated Assessment Models of the Food, Energy, and Water Nexus: A Review and an Outline of Research Needs , 2017 .

[39]  N. Kerr,et al.  Group performance and decision making. , 2004, Annual review of psychology.

[40]  Cees Bil,et al.  Concurrent Engineering Approaches for Sustainable Product Development in a Multi-Disciplinary Environment - Proceedings of the 19th ISPE International Conference on Concurrent Engineering (ISPE CE 2012), Trier, Germany, September 3-7, 2012 , 2013, ISPE CE.

[41]  Mario Carlo Severo,et al.  Goals matter: Amplification of the motivational significance of the feedback when goal impact is increased , 2018, Brain and Cognition.

[42]  Achim Zeileis,et al.  Diagnostic Checking in Regression Relationships , 2015 .

[43]  D. P. Thunnissen,et al.  Icemaker/sup TM/: an excel-based environment for collaborative design , 2003, 2003 IEEE Aerospace Conference Proceedings (Cat. No.03TH8652).

[44]  G. Stasser,et al.  Pooling of Unshared Information in Group Decision Making: Biased Information Sampling During Discussion , 1985 .

[45]  A. Zeileis.  Econometric Computing with HC and HAC Covariance Matrix Estimators , 2004 .

[46]  D. Bates,et al.  Fitting Linear Mixed-Effects Models Using lme4 , 2014, 1406.5823.