Can humans simulate talking like other humans? Comparing simulated clients to real customers in service inquiries

How authentic are inquiry calls made by simulated clients, or ‘mystery shoppers’, to service organizations, when compared to real callers? We analysed 48 simulated and 63 real inquiry calls to different veterinary practices in the United Kingdom and Ireland. The data were transcribed for conversation analysis, as well as coded for a variety of call categories including reason for the call, call outcome and turn design features. Analysis revealed systematic differences between real and simulated calls in terms of (1) reasons for the call, call outcome and call duration and (2) how callers refer to their pets in service requests and follow-up questions about their animal. Our qualitative analyses were supported with statistical summaries and tests. The findings reveal the limitations of mystery shopper methodology for the assessment of service provision. We also discuss the implications of the findings for the use of simulated encounters and the development of conversational agents.

[1]  B. Meltzer,et al.  Intelligent Machinery-National Physical Laboratory Report , 2011 .

[2]  J. Atkinson,et al.  A change-of-state token and aspects of its sequential placement , 1985 .

[3]  Galina B. Bolden,et al.  Transcribing for Social Research , 2017, QMiP Bulletin.

[4]  P. Calvert.  It's a mystery , 2005 .

[5]  R. Harwood,et al.  Using conversation analysis to inform role play and simulated interaction in communications skills training for healthcare professionals: identifying avenues for further development through a scoping review , 2018, BMC Medical Education.

[6]  Stuart Reeves,et al.  Some conversational challenges of talking with machines , 2017 .

[7]  John Skelton,et al.  The reality of role‐play: interruptions and amount of talk in simulated consultations , 2009, Medical education.

[8]  Emanuel A. Schegloff,et al.  Confirming Allusions: Toward an Empirical Account of Action , 1996, American Journal of Sociology.

[9]  Nicholas I. Fisher,et al.  Good and bad market research: A critical review of Net Promoter Score , 2018, Applied Stochastic Models in Business and Industry.

[10]  Brian Ott,et al.  The Limits of Control in Service Work: Interactive Routines and Interactional Competence , 2016 .

[11]  A. Mounsey,et al.  Do students develop better motivational interviewing skills through role‐play with standardised patients or with student colleagues? , 2006, Medical education.

[12]  E. Stokoe.  The (In)Authenticity of Simulated Talk: Comparing Role-Played and Actual Interaction and the Implications for Communication Training , 2013 .

[13]  Cees van der Vleuten,et al.  Students’ Views on the Use of Real Patients and Simulated Patients in Undergraduate Medical Education , 2009, Academic medicine : journal of the Association of American Medical Colleges.

[14]  E. Schegloff.  Some Practices for Referring to Persons in Talk-in-Interaction: A Partial Sketch of a Systematics , 1996 .

[15]  E. Goffman.  Frame analysis: An essay on the organization of experience , 1974 .

[16]  E. Schegloff.  The routine as achievement , 1986 .

[17]  P. Norris.  Purchasing restricted medicines in New Zealand pharmacies: results from a "mystery shopper" study , 2002, Pharmacy World and Science.

[18]  M. Sorjonen.  Constituting and maintaining activities across sequences: And-prefacing as a feature of question design , 1994, Language in Society.

[19]  A. Glasier,et al.  Community pharmacists providing emergency contraception give little advice about future contraceptive use: a mystery shopper study. , 2010, Contraception.

[20]  William Housley,et al.  In case of emergency, order pizza: an urgent case of action formation and recognition , 2019, CUI.

[21]  Alexa Hepburn,et al.  Crying: Notes on Description, Transcription, and Interaction , 2004 .

[22]  Steve Jacob,et al.  The mystery shopper: a tool to measure public service delivery? , 2018 .

[23]  Tanya Stivers,et al.  Coding Social Interaction: A Heretical Approach in Conversation Analysis? , 2015 .

[24]  Elizabeth Couper-Kuhlen,et al.  Constructing reason-for-the-call turns in everyday telephone conversation , 2001 .

[25]  S. Levinson.  Action formation and ascription , 2013 .

[26]  H. Garfinkel.  Studies in Ethnomethodology , 1968 .

[27]  Rein Ove Sikveland,et al.  The conversation analytic role-play method: simulation, endogenous impact and interactional nudges , 2016 .

[28]  J. Coulter.  What is “Discursive Psychology”? , 2004 .

[29]  Sarah Atkins.  Assessing health professionals’ communication through role-play: An interactional analysis of simulated versus actual general practice consultations , 2018, Discourse Studies.

[30]  E. Schegloff.  Some Other “Uh(m)”s , 2010 .

[31]  Alan Wilson,et al.  The role of mystery shopping in the measurement of service performance , 1998 .

[32]  T. Greenhalgh,et al.  Simulated consultations: a sociolinguistic perspective , 2016, BMC medical education.

[33]  H. H. Clark,et al.  Using uh and um in spontaneous speaking , 2002, Cognition.

[34]  J. D. de Ruiter,et al.  An Appeal for a Methodological Fusion of Conversation Analysis and Experimental Psychology , 2017 .

[35]  E. Schegloff,et al.  Opening up Closings , 1973 .

[36]  R. Owen.  Net Promoter Score and Its Successful Application , 2018, Marketing Wisdom.