Trust in Automation

Objective: We systematically review recent empirical research on factors that influence trust in automation to present a three-layered trust model that synthesizes existing knowledge. Background: Much of the existing research on factors that guide human-automation interaction is centered around trust, a variable that often determines the willingness of human operators to rely on automation. Studies have utilized a variety of different automated systems in diverse experimental paradigms to identify factors that impact operators’ trust. Method: We performed a systematic review of empirical research on trust in automation from January 2002 to June 2013. Papers were deemed eligible only if they reported the results of a human-subjects experiment in which humans interacted with an automated system in order to achieve a goal. Additionally, a relationship between trust (or a trust-related behavior) and another variable had to be measured. All together, 101 total papers, containing 127 eligible studies, were included in the review. Results: Our analysis revealed three layers of variability in human–automation trust (dispositional trust, situational trust, and learned trust), which we organize into a model. We propose design recommendations for creating trustworthy automation and identify environmental conditions that can affect the strength of the relationship between trust and reliance. Future research directions are also discussed for each layer of trust. Conclusion: Our three-layered trust model provides a new lens for conceptualizing the variability of trust in automation. Its structure can be applied to help guide future research and develop training interventions and design procedures that encourage appropriate trust.

[1]  René Riedl,et al.  Are There Neural Gender Differences in Online Trust? An fMRI Study on the Perceived Trustworthiness of eBay Offers , 2010, MIS Q..

[2]  Stephen Marsh,et al.  The role of trust in information science and technology , 2005, Annu. Rev. Inf. Sci. Technol..

[3]  TerryAnn Glandon,et al.  Framing, decision-aid systems, and culture: Exploring influences on fraud investigations , 2012, Int. J. Account. Inf. Syst..

[4]  Christopher S. Campbell,et al.  Deciding when to trust automation in a policy-based city management game: policity , 2007, CHIMIT '07.

[5]  D. H. Mills.  The Logic and Limits of Trust , 1983 .

[6]  John Richardson,et al.  Alarm timing, trust and driver expectation for forward collision warning systems. , 2006, Applied ergonomics.

[7]  Bako Rajaonah,et al.  The role of intervening variables in driver-ACC cooperation , 2008, Int. J. Hum. Comput. Stud..

[8]  Morton Deutschi.  The Effect of Motivational Orientation upon Trust and Suspicion , 1960 .

[9]  R. Riedl,et al.  The Biology of Trust: Integrating Evidence From Genetics, Endocrinology, and Functional Brain Imaging , 2012 .

[10]  Deborah Lee,et al.  I Trust It, but I Don’t Know Why , 2013, Hum. Factors.

[11]  Arthur D. Fisk,et al.  Understanding the Effect of Workload on Automation Use for Younger and Older Adults , 2011, Hum. Factors.

[12]  Fang-Wu Tung,et al.  Influence of Gender and Age on the Attitudes of Children towards Humanoid Robots , 2011, HCI.

[13]  Christopher D. Wickens,et al.  On the Independence of Compliance and Reliance: Are Automation False Alarms Worse Than Misses? , 2007, Hum. Factors.

[14]  Raja Parasuraman,et al.  Transitioning to Future Air Traffic Management: Effects of Imperfect Automation on Controller Attention and Performance , 2010, Hum. Factors.

[15]  Brent A. Scott,et al.  Trust, trustworthiness, and trust propensity: a meta-analytic test of their unique relationships with risk taking and job performance. , 2007, The Journal of applied psychology.

[16]  Ericka Rovira,et al.  Understanding reliance on automation: effects of error type, error distribution, age and experience , 2014, Theoretical issues in ergonomics science.

[17]  Gregg H. Gunsch,et al.  The effect of external safeguards on human-information system trust in an information warfare environment , 2001, 36th Annual Hawaii International Conference on System Sciences, 2003. Proceedings of the.

[18]  Poornima Madhavan,et al.  The effect of distractor modality and processing code on human-automation interaction , 2011, Cognition, Technology & Work.

[19]  Greg A. Jamieson,et al.  The impact of context-related reliability on automation failure detection and scanning behaviour , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[20]  John Yen,et al.  The influence of agent reliability on trust in human-agent collaboration , 2008, ECCE '08.

[21]  Poornima Madhavan,et al.  Effects of computer self-efficacy and system reliability on user interaction with decision support systems , 2010, Comput. Hum. Behav..

[22]  Arthur D. Fisk,et al.  Type of Automation Failure: The Effects on Trust and Reliance in Automation , 2004 .

[23]  Douglas A. Wiegmann,et al.  Effects of Information Source, Pedigree, and Reliability on Operator Interaction With Decision Support Systems , 2007, Hum. Factors.

[24]  Andrew K. Mayer,et al.  The manipulation of user expectancies: effects on reliance, compliance, and trust using an automated system , 2008 .

[25]  Christopher D. Wickens,et al.  A model for types and levels of human interaction with automation , 2000, IEEE Trans. Syst. Man Cybern. Part A.

[26]  A. Fisk,et al.  Reliability and Age-Related Effects on Trust and Reliance of a Decision Support Aid , 2004 .

[27]  Brian Daniel Green.  Applying human characteristics of trust to animated anthropomorphic software agents , 2010 .

[28]  Jessie Y. C. Chen,et al.  A Meta-Analysis of Factors Affecting Trust in Human-Robot Interaction , 2011, Hum. Factors.

[29]  Karen Zita Haigh,et al.  A Model of Trust and Reliance of Automation Technology for Older Users , 2005, AAAI Fall Symposium: Caring Machines.

[30]  Cees J. H. Midden,et al.  Effect of indirect information on system trust and control allocation , 2008, Behav. Inf. Technol..

[31]  Dietrich Manzey,et al.  Misuse of Automated Aids in Process Control: Complacency, Automation Bias and Possible Training Interventions , 2006 .

[32]  M. David Keller,et al.  The Effect of Gestalt Psychology on the System-Wide Trust Strategy in Automation , 2011 .

[33]  Arthur D. Fisk,et al.  Reliance on Automation as a Function of Expectation of Reliability, Cost of Verification, and Age , 2007 .

[34]  Raja Parasuraman,et al.  Effects of Imperfect Automation on Decision Making in a Simulated Command and Control Task , 2007, Hum. Factors.

[35]  Julian Sanchez,et al.  FACTORS THAT AFFECT TRUST AND RELIANCE ON AN AUTOMATED AID , 2006 .

[36]  Daniel Gopher,et al.  Effect of Descriptive Information and Experience on Automation Reliance , 2011, Hum. Factors.

[37]  Christopher D. Wickens,et al.  Management of Multiple Uavs with Imperfect Automation , 2006 .

[38]  Ernesto A. Bustamante,et al.  Effects of False-Alarm vs. Miss-Prone Automation and Likelihood Alarm Technology on Trust, Reliance, and Compliance in a Miss-Prone Task , 2010 .

[39]  Raja Parasuraman,et al.  Adaptive Aiding of Human-Robot Teaming , 2011 .

[40]  Charles T. Scialfa,et al.  Age differences in trust and reliance of a medication management system , 2005, Interact. Comput..

[41]  Arthur D. Fisk,et al.  Effects of Mental Model Quality on Collaborative System Performance , 2007 .

[42]  Li Gong,et al.  How social is social responses to computers? The function of the degree of anthropomorphism in computer representations , 2008, Comput. Hum. Behav..

[43]  Justin G. Hollands,et al.  Beyond Identity: Incorporating System Reliability Information Into an Automated Combat Identification System , 2011, Hum. Factors.

[44]  Stephen R. Dixon.  Imperfect Diagnostic Automation: How Adjusting Bias and Saliency Affects Operator Trust , 2006 .

[45]  John D. Lee,et al.  Trust in Automation: Designing for Appropriate Reliance , 2004, Hum. Factors.

[46]  Ann M. Bisantz,et al.  The impact of cognitive feedback on judgment performance and trust with decision aids , 2008 .

[47]  Clifford Nass,et al.  Computers are social actors , 1994, CHI '94.

[48]  Mark W. Scerbo,et al.  Automation-induced complacency for monitoring highly reliable systems: the role of task complexity, system experience, and operator trust , 2007 .

[49]  Lu Wang,et al.  The Effects of Design Features on Users’ Trust in and Reliance on a Combat Identification System , 2011 .

[50]  Christopher D. Wickens,et al.  Effects of Age on Utilization and Perceived Reliability of an Automated Decision-Making aid for Luggage Screening , 2003 .

[51]  D. Wiegmann,et al.  Similarities and differences between human–human and human–automation trust: an integrative review , 2007 .

[52]  Mary L. Cummings,et al.  The Role of Human-Automation Consensus in Multiple Unmanned Vehicle Scheduling , 2010, Hum. Factors.

[53]  Yung-Ming Li,et al.  Increasing trust in mobile commerce through design aesthetics , 2010, Comput. Hum. Behav..

[54]  Jinwoo Kim,et al.  Designing towards emotional usability in customer interfaces trustworthiness of cyber-banking system interfaces , 1998, Interact. Comput..

[55]  Diane M. Strong,et al.  EXTENDING THE TASK-TECHNOLOGY FIT MODEL WITH SELF-EFFICACY CONSTRUCTS , 2002 .

[56]  Dietrich Manzey,et al.  Human Performance Consequences of Automated Decision Aids in States of Sleep Loss , 2011, Hum. Factors.

[57]  Bernt Schiele,et al.  Towards improving trust in context-aware systems by displaying system confidence , 2005, Mobile HCI.

[58]  Kim,et al.  Social conflict: Escalation, stalemate, and settlement , 1986 .

[59]  Mark A. Daly.  Task Load and Automation Use in an Uncertain Environment , 2012 .

[60]  John D. Lee,et al.  Effect of Shared Information on Trust and Reliance in a Demand Forecasting Task , 2006 .

[61]  Mark R. Lehto,et al.  Foundations for an Empirically Determined Scale of Trust in Automated Systems , 2000 .

[62]  J. G. Holmes,et al.  Trust in close relationships. , 1985 .

[63]  Christopher A. Miller,et al.  Trust and etiquette in high-criticality automated systems , 2004, CACM.

[64]  Joseph B. Lyons,et al.  Human–Human Reliance in the Context of Automation , 2012, Hum. Factors.

[65]  J. H. Davis,et al.  An Integrative Model Of Organizational Trust , 1995 .

[66]  Tobias Fritsch,et al.  A Systematic Review of Trust in Automation and Assistance Systems for Older Persons’ Overall Requirements , 2012, eTELEMED 2012.

[67]  Tao Zhou,et al.  The effect of initial trust on user adoption of mobile payment , 2011 .

[68]  Pei-Luen Patrick Rau,et al.  A Cross-cultural Study: Effect of Robot Appearance and Task , 2010, Int. J. Soc. Robotics.

[69]  Janet E. Miller,et al.  Designing for Human-Centered Systems: Situational Risk as a Factor of Trust in Automation , 2010 .

[70]  John Richardson,et al.  The effect of alarm timing on driver behaviour: an investigation of differences in driver trust and response to alarms according to alarm timing , 2004 .

[71]  Patricia Delhomme,et al.  Simulator Training With a Forward Collision Warning System , 2012, Hum. Factors.

[72]  Linda G. Pierce,et al.  Automation Usage Decisions: Controlling Intent and Appraisal Errors in a Target Detection Task , 2007, Hum. Factors.

[73]  Jeffrey M. Bradshaw,et al.  The Dynamics of Trust in Cyberdomains , 2009, IEEE Intelligent Systems.

[74]  Randall D. Spain.  The Effects of Automation Expertise, System Confidence, and Image Quality on Trust, Compliance, and Performance , 2009 .

[75]  Douglas A. Wiegmann,et al.  PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 49th ANNUAL MEETING—2005 487 EFFECTS OF INFORMATION SOURCE, PEDIGREE, AND RELIABILITY ON OPERATORS ’ UTILIZATON OF DIAGNOSTIC ADVICE , 2022 .

[76]  Anthony J. Masalonis.  Effects of training operators on situation-specific automation reliability , 2003, SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme - System Security and Assurance (Cat. No.03CH37483).

[77]  N Moray,et al.  Trust, control strategies and allocation of function in human-machine systems. , 1992, Ergonomics.

[78]  Tatsuya Nomura,et al.  Prediction of Human Behavior in Human--Robot Interaction Using Psychological Scales for Anxiety and Negative Attitudes Toward Robots , 2008, IEEE Transactions on Robotics.

[79]  David B. Kaber,et al.  The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task , 2004 .

[80]  Cees J. H. Midden,et al.  The effects of errors on system trust, self-confidence, and the allocation of control in route planning , 2003, Int. J. Hum. Comput. Stud..

[81]  James L. Szalma,et al.  The Effect of Automation Reliability on User Automation Trust and Reliance in a Search-and-Rescue Scenario , 2008 .

[82]  Jean-Marc Robert,et al.  Trust in new decision aid systems , 2006, IHM '06.

[83]  Stephen Rice.  Examining Single- and Multiple-Process Theories of Trust in Automation , 2009, The Journal of general psychology.

[84]  Nadine B. Sarter,et al.  Supporting Trust Calibration and the Effective Use of Decision Aids by Presenting Dynamic System Confidence Information , 2006, Hum. Factors.

[85]  Janine Willis,et al.  First Impressions , 2006, Psychological science.

[86]  Tobias Felsing,et al.  Cognitive diversity and team performance in a complex multiple task environment , 2006, Ergonomics.

[87]  S. Czaja,et al.  Age differences in attitudes toward computers. , 1998, The journals of gerontology. Series B, Psychological sciences and social sciences.

[88]  Richard Pak,et al.  Decision support aids with anthropomorphic characteristics influence trust and performance in younger and older adults , 2012, Ergonomics.

[89]  Ben Willems.  Decision Support Automation Research in the En Route Air Traffic Control Environment , 2002 .

[90]  Choon-Ling Sia,et al.  Consumer trust and distrust: An issue of website design , 2010, Int. J. Hum. Comput. Stud..

[91]  Kenneth Littlejohn,et al.  Accounting for the human in cyberspace: Effects of mood on trust in automation , 2010, 2010 International Symposium on Collaborative Technologies and Systems.

[92]  Andrew D. Engell,et al.  Implicit Trustworthiness Decisions: Automatic Coding of Face Properties in the Human Amygdala , 2007, Journal of Cognitive Neuroscience.

[93]  Alison Parkes,et al.  Persuasive Decision Support: Improving Reliance on Decision Support Systems , 2009, PACIS.

[94]  J. Rotter.  A new scale for the measurement of interpersonal trust. , 1967, Journal of personality.

[95]  Lu Wang,et al.  Developing Human-Machine Interfaces to Support Appropriate Trust and Reliance on Automated Combat Identification Systems (Developpement d'Interfaces Homme-Machine Pour Appuyer la Confiance dans les Systemes Automatises d'Identification au Combat) , 2007 .

[96]  Raja Parasuraman,et al.  Humans and Automation: Use, Misuse, Disuse, Abuse , 1997, Hum. Factors.

[97]  Arthur D. Fisk,et al.  Don't Let Me Down: The Role of Operator Expectations in Human-Automation Interaction , 2006 .

[98]  Lu Wang,et al.  Trust and Reliance on an Automated Combat Identification System , 2009, Hum. Factors.

[99]  Victor A. Riley,et al.  Operator reliance on automation: Theory and data. , 1996 .

[100]  D. Mohr,et al.  Harnessing Context Sensing to Develop a Mobile Intervention for Depression , 2011, Journal of medical Internet research.

[101]  Michael J. Prietula,et al.  The Turing effect: the nature of trust in expert systems advice , 1997 .

[102]  Andrea M. Cassidy.  Mental Models, Trust, and Reliance: Exploring the Effect of Human Perceptions on Automation Use , 2009 .

[103]  David P. Biros,et al.  The Influence of Task Load and Automation Trust on Deception Detection , 2004 .

[104]  Randall D. Spain,et al.  The Role of Automation Etiquette and Pedigree in Trust and Dependence , 2009 .

[105]  Linda Onnasch,et al.  Human Performance Consequences of Automated Decision Aids , 2012 .

[106]  Susan Wiedenbeck,et al.  On-line trust: concepts, evolving themes, a model , 2003, Int. J. Hum. Comput. Stud..

[107]  Linda G. Pierce,et al.  Effects of Human—Machine Competition on Intent Errors in a Target Detection Task , 2009, Hum. Factors.

[108]  Monica N. Lees,et al.  The influence of distraction and driving context on driver response to imperfect collision warning systems , 2007, Ergonomics.

[109]  Eun-Ju Lee,et al.  Flattery may get computers somewhere, sometimes: The moderating role of output modality, computer gender, and user gender , 2008, Int. J. Hum. Comput. Stud..

[110]  Linda G. Pierce,et al.  The Influence of Feedback on Automation Use, Misuse, and Disuse , 2002 .

[111]  J. Rotter.  Interpersonal trust, trustworthiness, and gullibility. , 1980 .

[112]  Michael Workman,et al.  Expert decision support system use, disuse, and misuse: a study using the theory of planned behavior , 2005, Comput. Hum. Behav..

[113]  Angelika Dimoka,et al.  What Does the Brain Tell Us About Trust and Distrust? Evidence from a Functional Neuroimaging Study , 2010, MIS Q..

[114]  E. Bass,et al.  The Effect of Information Analysis Automation Display Content on Human Judgment Performance in Noisy Environments , 2013, Journal of cognitive engineering and decision making.

[115]  Arthur D. Fisk,et al.  Do Younger and Older Adults Differentially Depend on an Automated System? , 2010 .

[116]  Christopher D. Wickens,et al.  Automation Reliability in Unmanned Aerial Vehicle Control: A Reliance-Compliance Model of Automation Dependence in High Workload , 2006, Hum. Factors.

[117]  Lemuria Carter,et al.  The impact of personality on nurses' bias towards automated decision aid acceptance , 2012, Int. J. Inf. Syst. Chang. Manag..

[118]  Douglas A. Wiegmann,et al.  Cognitive Anchoring on Self-Generated Decisions Reduces Operator Reliance on Automated Diagnostic Aids , 2005, Hum. Factors.

[119]  Pei-Luen Patrick Rau,et al.  Effects of communication style and culture on ability to accept recommendations from robots , 2009, Comput. Hum. Behav..

[120]  Nash Stanton,et al.  The Effects of System Technology and Probability Type on Trust, Compliance, and Reliance , 2009 .

[121]  J. M. Ross.  Moderators Of Trust And Reliance Across Multiple Decision Aids , 2008 .

[122]  Robin L. Wakefield,et al.  The Role of Web Site Characteristics in Initial Trust Formation , 2004, J. Comput. Inf. Syst..

[123]  Tal Oron-Gilad,et al.  The effect of system aesthetics on trust, cooperation, satisfaction and annoyance in an imperfect automated system. , 2012, Work.

[124]  Igor Dolgov,et al.  The Influence of Perceived Task Difficulty and Importance on Automation Use , 2010 .

[125]  Daniel R. Ilgen,et al.  Not All Trust Is Created Equal: Dispositional and History-Based Trust in Human-Automation Interactions , 2008, Hum. Factors.

[126]  Stephanie M. Merritt.  Affective Processes in Human–Automation Interactions , 2011, Hum. Factors.

[127]  Ann M. Bisantz,et al.  Assessment of operator trust in and utilization of automated decision-aids under different framing conditions , 2001 .

[128]  Douglas A. Wiegmann,et al.  Automation Failures on Tasks Easily Performed by Operators Undermines Trust in Automated Aids , 2003 .

[129]  Detmar W. Straub,et al.  Trust and TAM in Online Shopping: An Integrated Model , 2003, MIS Q..

[130]  Raja Parasuraman,et al.  The World is not Enough: Trust in Cognitive Agents , 2012 .

[131]  Grant S. Taylor,et al.  Individual differences in response to automation: the five factor model of personality. , 2011, Journal of experimental psychology. Applied.

[132]  Arthur D. Fisk,et al.  Age-Related Differences in Reliance Behavior Attributable to Costs Within a Human-Decision Aid System , 2008, Hum. Factors.

[133]  Ting-Peng Liang.  Cognitive Neuroscience in Information Systems Research , 2012, Pac. Asia J. Assoc. Inf. Syst..

[134]  A W Stedmon,et al.  Semi-automated CCTV surveillance: the effects of system confidence, system accuracy and task complexity on operator vigilance, reliance and workload. , 2013, Applied ergonomics.

[135]  Bobbie D. Seppelt,et al.  Making adaptive cruise control (ACC) limits visible , 2007, Int. J. Hum. Comput. Stud..

[136]  C. Nass,et al.  Are People Polite to Computers? Responses to Computer-Based Interviewing Systems1 , 1999 .

[137]  Regina A. Pomranky,et al.  The role of trust in automation reliance , 2003, Int. J. Hum. Comput. Stud..

[138]  Douglas A. Wiegmann,et al.  Effects of Attribute and Goal Framing on Automation Reliance and Compliance , 2005 .

[139]  Eric N. Wiebe,et al.  The Effects of Automated Decision Algorithm Modality and Transparency on Reported Trust and Task Performance , 2008 .