Tell me more! Assessing interactions with social robots from speech

Abstract As social robots are increasingly introduced into health interventions, one potential area where they might prove valuable is in supporting people’s psychological health through conversation. Given the importance of self-disclosure for psychological health, this study assessed the viability of using social robots for eliciting rich disclosures that identify needs and emotional states in human interaction partners. Three within-subject experiments were conducted with participants interacting with another person, a humanoid social robot, and a disembodied conversational agent (voice assistant). We performed a number of objective evaluations of disclosures to these three agents via speech content and voice analyses and also probed participants’ subjective evaluations of their disclosures to three agents. Our findings suggest that participants overall disclose more to humans than artificial agents, that agents’ embodiment influences disclosure quantity and quality, and that people are generally aware of differences in their personal disclosures to three agents studied here. Together, the findings set the stage for further investigation into the psychological underpinnings of self-disclosures to artificial agents and their potential role in eliciting disclosures as part of mental and physical health interventions.

[1]  Gabriel Skantze,et al.  User feedback in human-robot interaction: prosody, gaze and timing , 2013, INTERSPEECH.

[2]  J. Omarzu A Disclosure Decision Model: Determining How and When Individuals Will Self-Disclose , 2000 .

[3]  Elin A. Björling,et al.  Sharing Stress With a Robot: What Would a Robot Say? , 2020 .

[4]  Kerstin Fischer,et al.  Speech Melody Matters—How Robots Profit from Using Charismatic Speech , 2019 .

[5]  Bruce A. MacDonald,et al.  Robots in Older People's Homes to Improve Medication Adherence and Quality of Life: A Randomised Cross-Over Trial , 2014, ICSR.

[6]  Jan B. F. van Erp,et al.  Social Touch in Human–Robot Interaction: Robot-Initiated Touches can Induce Positive Responses without Extensive Prior Bonding , 2018, Int. J. Soc. Robotics.

[7]  T. Shibata,et al.  Use of a Therapeutic, Socially Assistive Pet Robot (PARO) in Improving Mood and Stimulating Social Interaction and Communication for People With Dementia: Study Protocol for a Randomized Controlled Trial , 2015, JMIR research protocols.

[8]  Brian Scassellati,et al.  Prosody recognition in male infant-directed speech , 2004, 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04CH37566).

[9]  Guy Laban,et al.  Working Together with Conversational Agents: The Relationship of Perceived Cooperation with Service Performance Evaluations , 2019, CONVERSATIONS.

[10]  Emily S. Cross,et al.  Tell me more! Assessing interactions with social robots from speech , 2020, Paladyn J. Behav. Robotics.

[11]  H. Ishiguro,et al.  The impact of robotic intervention on joint attention in children with autism spectrum disorders , 2018, Molecular Autism.

[12]  J. Borod,et al.  A Meta-Analysis of the Effects of Written Emotional Disclosure on the Health Outcomes of Clinical Populations , 2004, The Journal of nervous and mental disease.

[13]  Hamutal Kreiner,et al.  What You Say and How You Say It , 2016 .

[14]  Leif D. Nelson,et al.  Let's Publish Fewer Papers , 2012 .

[15]  Annica Kristoffersson,et al.  The Relevance of Social Cues in Assistive Training with a Social Robot , 2018, ICSR.

[16]  Brian Scassellati,et al.  Secret-sharing: Interactions between a child, robot, and adult , 2011, 2011 IEEE International Conference on Systems, Man, and Cybernetics.

[17]  Guy Laban,et al.  The Effect of Personalization Techniques in Users' Perceptions of Conversational Recommender Systems , 2020, IVA.

[18]  C. Antaki,et al.  Diagnostic formulations in psychotherapy , 2005 .

[19]  Richard L. Daft,et al.  Organizational information requirements, media richness and structural design , 1986 .

[20]  D. Bem Self-Perception Theory , 1972 .

[21]  Guy Laban,et al.  Let's Talk About It! Subjective and Objective Disclosures to Social Robots , 2020, HRI.

[22]  Elin A. Björling,et al.  Can We Keep Him Forever? Teens’ Engagement and Desire for Emotional Connection with a Social Robot , 2020, Int. J. Soc. Robotics.

[23]  P. Roach,et al.  Transcription of Prosodic and Paralinguistic Features of Emotional Speech , 1998, Journal of the International Phonetic Association.

[24]  Ho Seok Ahn,et al.  Smiling and use of first-name by a healthcare receptionist robot: Effects on user perceptions, attitudes, and behaviours , 2020, Paladyn J. Behav. Robotics.

[25]  Craig J. Sutherland,et al.  The Effect of Robot Attentional Behaviors on User Perceptions and Behaviors in a Simulated Health Care Interaction: Randomized Controlled Trial , 2019, Journal of medical Internet research.

[26]  Hideyuki Nakanishi,et al.  “That Robot Stared Back at Me!”: Demonstrating Perceptual Ability Is Key to Successful Human–Robot Interactions , 2019, Front. Robot. AI.

[27]  M. Sherif,et al.  The psychology of attitudes. , 1946, Psychological review.

[28]  Shelly L. Gable,et al.  What do you do when things go right? The intrapersonal and interpersonal benefits of sharing positive events. , 2004, Journal of personality and social psychology.

[29]  Mario Bunge,et al.  A General Black Box Theory , 1963, Philosophy of Science.

[30]  M. Mahoney Publication prejudices: An experimental study of confirmatory bias in the peer review system , 1977, Cognitive Therapy and Research.

[31]  Guy Hoffman,et al.  What robots can teach us about intimacy: The reassuring effects of robot responsiveness to human disclosure , 2016, Comput. Hum. Behav..

[32]  J. Kahn,et al.  The Distress Disclosure Index: a research review and multitrait-multimethod examination. , 2012, Journal of counseling psychology.

[33]  Shanyang Zhao,et al.  Humanoid social robots as a medium of communication , 2006, New Media Soc..

[34]  J. Pittam Voice in Social Interaction: An Interdisciplinary Approach , 1994 .

[35]  M. Brass,et al.  Believing androids – fMRI activation in the right temporo-parietal junction is modulated by ascribing intentions to non-human agents , 2017, Social neuroscience.

[36]  Marian S. Harris,et al.  Self-disclosure reciprocity, liking and the deviant , 1973 .

[37]  M. Patterson Compensation in Nonverbal Immediacy Behaviors: A Review , 1973 .

[38]  Agnieszka Wykowska,et al.  From social brains to social robots: applying neurocognitive insights to human–robot interaction , 2019, Philosophical Transactions of the Royal Society B.

[39]  Rosalind W. Picard,et al.  Stress measurement using speech: Recent advancements, validation issues, and ethical and privacy considerations , 2019, Stress.

[40]  V. Derlega,et al.  Clients' Responses to Dissatisfaction in Psychotherapy: A Test of Rusbult's Exit-Voice-Loyalty-Neglect Model , 1993 .

[41]  Cynthia Breazeal,et al.  A Social Robot to Mitigate Stress, Anxiety, and Pain in Hospital Pediatric Care , 2015, HRI.

[42]  K. Scherer,et al.  Handbook of affective sciences. , 2003 .

[43]  R. Frick Communicating emotion: The role of prosodic features. , 1985 .

[44]  P. Sterkenburg,et al.  Robot‐mediated therapy to reduce worrying in persons with visual and intellectual disabilities , 2020, Journal of applied research in intellectual disabilities : JARID.

[45]  John P. A. Ioannidis,et al.  A manifesto for reproducible science , 2017, Nature Human Behaviour.

[46]  Guy Hoffman,et al.  Machines as a source of consolation: Robot responsiveness increases human approach behavior and desire for companionship , 2016, 2016 11th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[47]  S. Boll,et al.  Suppression of acoustic noise in speech using spectral subtraction , 1979 .

[48]  D. Crystal Systems of prosodic and paralinguistic features in English / by David Crystall and Randolph Quirk , 1964 .

[49]  John O. Greene A cognitive approach to human communication: An action assembly theory , 1984 .

[50]  Paul Boersma,et al.  Praat, a system for doing phonetics by computer , 2002 .

[51]  J. Tangney,et al.  Social psychological foundations of clinical psychology , 2010 .

[52]  Jennifer byrd-craven,et al.  Vocal indices of stress: a review. , 2013, Journal of voice : official journal of the Voice Foundation.

[53]  M. Mason,et al.  Mind Perception , 2021, Encyclopedia of Evolutionary Psychological Science.

[54]  Lawrence A. Hosman The evaluational consequences of topic reciprocity and self‐disclosure reciprocity , 1987 .

[55]  Norihiro Hagita,et al.  Robot Reciprocation of Hugs Increases Both Interacting Times and Self-disclosures , 2020, International Journal of Social Robotics.

[56]  G. Metta,et al.  Robots As Intentional Agents: Using Neuroscientific Methods to Make Robots Appear More Social , 2017, Front. Psychol..

[57]  G. Saucier Mini-markers: a brief version of Goldberg's unipolar big-five markers. , 1994, Journal of personality assessment.

[58]  Self Book Self Disclosure An Experimental Analysis Of The Transparent Self , 2016 .

[59]  Emily S. Cross,et al.  The shaping of social perception by stimulus and knowledge cues to human animacy , 2016, Philosophical Transactions of the Royal Society B: Biological Sciences.

[60]  M. Maheu,et al.  A Review of Telepresence, Virtual Reality, and Augmented Reality Applied to Clinical Care , 2020, Journal of Technology in Behavioral Science.

[61]  J. Frattaroli Experimental disclosure and its moderators: a meta-analysis. , 2006, Psychological bulletin.

[62]  Eric Gilbert,et al.  VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text , 2014, ICWSM.

[63]  Cynthia Breazeal,et al.  Older Adults Living With Social Robots: Promoting Social Connectedness in Long-Term Communities , 2019, IEEE Robotics & Automation Magazine.

[64]  Jeffrey F. Cohn,et al.  Detecting Depression Severity from Vocal Prosody , 2013, IEEE Transactions on Affective Computing.

[65]  C. Berger,et al.  Language and social knowledge: Uncertainty in interpersonal relations , 1982 .

[66]  Emily S. Cross,et al.  From automata to animate beings: the scope and limits of attributing socialness to artificial agents , 2018, Annals of the New York Academy of Sciences.

[67]  Ronit Feingold Polak,et al.  Social Robot for Rehabilitation: Expert Clinicians and Post-Stroke Patients' Evaluation Following a Long-Term Intervention , 2020 .

[68]  Rosa María Baños,et al.  Positive Technology: Using Interactive Technologies to Promote Positive Functioning , 2012, Cyberpsychology Behav. Soc. Netw..

[69]  M. Argyle,et al.  EYE-CONTACT, DISTANCE AND AFFILIATION. , 1965, Sociometry.

[70]  John R. Carlson,et al.  Channel Expansion Theory and the Experiential Nature of Media Richness Perceptions , 1999 .

[71]  Blockin,et al.  Vocal Expression of Emotion , 2004 .

[72]  Adam S. Miner,et al.  Psychological, Relational, and Emotional Effects of Self-Disclosure After Conversations With a Chatbot , 2018, The Journal of communication.

[73]  J. Pennebaker,et al.  The Psychological Meaning of Words: LIWC and Computerized Text Analysis Methods , 2010 .

[74]  A. Gaggioli,et al.  Digital Mental Health Tools for Caregivers of Older Adults—A Scoping Review , 2020, Frontiers in Public Health.

[75]  Bart de Boer,et al.  Introducing Parselmouth: A Python interface to Praat , 2018, J. Phonetics.

[76]  John O. Greene Production of Messages in Pursuit of Multiple Social Goals: Action Assembly Theory Contributions to the Study of Cognitive Encoding Processes , 1995 .

[77]  Cynthia Breazeal,et al.  Recognition of Affective Communicative Intent in Robot-Directed Speech , 2002, Auton. Robots.

[78]  A. Loutfi,et al.  The Influence of Feedback Type in Robot-Assisted Training , 2019, Multimodal Technol. Interact..

[79]  Giulio Sandini,et al.  Social Cognition for Human-Robot Symbiosis—Challenges and Building Blocks , 2018, Front. Neurorobot..

[80]  Emily S. Cross,et al.  The Perception of Emotion in Artificial Agents , 2018, IEEE Transactions on Cognitive and Developmental Systems.

[81]  S. Chaiken Heuristic versus systematic information processing and the use of source versus message cues in persuasion. , 1980 .

[82]  Nadia Magnenat-Thalmann,et al.  Body Movements Generation for Virtual Characters and Social Robots , 2017, Social Signal Processing.

[83]  E. Broadbent Interactions With Robots: The Truths We Reveal About Ourselves , 2017, Annual review of psychology.

[84]  Emily S. Cross,et al.  The Control of Automatic Imitation Based on Bottom–Up and Top–Down Cues to Animacy: Insights from Brain and Behavior , 2014, Journal of Cognitive Neuroscience.

[85]  Erin D. Reilly,et al.  Use of Social Robots in Mental Health and Well-Being Research: Systematic Review , 2019, Journal of medical Internet research.

[86]  A Guell,et al.  Voice analysis to predict the psychological or physical state of a speaker. , 1990, Aviation, space, and environmental medicine.

[87]  D. Schacter The seven sins of memory. Insights from psychology and cognitive neuroscience. , 1999, The American psychologist.

[88]  Emilia I. Barakova,et al.  Game-Based Human-Robot Interaction Promotes Self-disclosure in People with Visual Impairments and Intellectual Disabilities , 2019, IWINAC.

[89]  Kurt Gray,et al.  Mind Perception Is the Essence of Morality , 2012, Psychological inquiry.

[90]  S. Chaiken,et al.  Motivated Heuristic and Systematic Processing , 1999 .

[91]  Guy Hoffman,et al.  Robot Responsiveness to Human Disclosure Affects Social Impression and Appeal , 2014, 2014 9th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[92]  Johan F. Hoorn,et al.  Theory of Robot Communication: I. The Medium is the Communication Partner , 2018, Int. J. Humanoid Robotics.

[93]  D. Bem Self-perception: An alternative interpretation of cognitive dissonance phenomena. , 1967, Psychological review.

[94]  Xavier Neyt,et al.  Voice Stress Analysis: A New Framework for Voice and Effort in Human Performance , 2018, Front. Psychol..

[95]  H. Kreiner,et al.  Self-Disclosure Here and Now: Combining Retrospective Perceived Assessment With Dynamic Behavioral Measures , 2019, Front. Psychol..

[96]  H. Beek F1000Prime recommendation of False-positive psychology: undisclosed flexibility in data collection and analysis allows presenting anything as significant. , 2012 .

[97]  Yael Edan,et al.  Differences between Young and Old Users when Interacting with a Humanoid Robot: A Qualitative Usability Study , 2018, HRI.

[98]  C. Berger,et al.  SOME EXPLORATIONS IN INITIAL INTERACTION AND BEYOND: TOWARD A DEVELOPMENTAL THEORY OF INTERPERSONAL COMMUNICATION , 1975 .

[99]  Ruth Jepson,et al.  Six steps in quality intervention development (6SQuID) , 2015, Journal of Epidemiology & Community Health.

[100]  I. Firestone,et al.  Reconciling verbal and nonverbal models of dyadic communication , 1977 .

[101]  Ruben C. Arslan,et al.  formr: A study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R , 2018, Behavior Research Methods.

[102]  Brian Scassellati,et al.  Vulnerable robots positively shape human conversational dynamics in a human–robot team , 2020, Proceedings of the National Academy of Sciences.

[103]  A. Tversky,et al.  Subjective Probability: A Judgment of Representativeness , 1972 .

[104]  Brian Scassellati,et al.  How people talk when teaching a robot , 2009, 2009 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[105]  Peter Carruthers,et al.  The illusion of conscious will , 2007, Synthese.

[106]  D. Wegner,et al.  Dimensions of Mind Perception , 2007, Science.

[107]  H. Ishiguro,et al.  Can Robotic Systems Promote Self-Disclosure in Adolescents with Autism Spectrum Disorder? A Pilot Study , 2018, Front. Psychiatry.

[108]  B. MacDonald,et al.  Attitudes towards health‐care robots in a retirement village , 2012, Australasian journal on ageing.

[109]  J. Grimshaw,et al.  Methods for designing interventions to change healthcare professionals’ behaviour: a systematic review , 2017, Implementation Science.

[110]  B. MacDonald,et al.  The psychosocial effects of a companion robot: a randomized controlled trial. , 2013, Journal of the American Medical Directors Association.

[111]  Jeanne C. Watson,et al.  How and When Does Emotional Expression Help? , 2001 .

[112]  Benjamin Weiss Talker Quality in Human and Machine Interaction: Modeling the Listener’s Perspective in Passive and Interactive Scenarios , 2020 .

[113]  Cynthia Breazeal,et al.  Toward sociable robots , 2003, Robotics Auton. Syst..

[114]  Norihiro Hagita,et al.  How Do Communication Cues Change Impressions of Human–Robot Touch Interaction? , 2018, Int. J. Soc. Robotics.

[115]  Panos Markopoulos,et al.  How do People Perceive Privacy and Interaction Quality while Chatting with a Crowd-operated Robot? , 2020, HRI.

[116]  Giulio Sandini,et al.  Trust and Social Engineering in Human Robot Interaction: Will a Robot Make You Disclose Sensitive Information, Conform to Its Recommendations or Gamble? , 2018, IEEE Robotics and Automation Letters.

[117]  T. Kamarck,et al.  A global measure of perceived stress. , 1983, Journal of health and social behavior.

[118]  Yun Huang,et al.  "I Hear You, I Feel You": Encouraging Deep Self-disclosure through a Chatbot , 2020, CHI.