“How am I Doing?”: A New Framework to Effectively Measure the Performance of Automated Customer Care Contact Centers

Satisfying callers’ goals and expectations is the primary objective of every customer care contact center. However, quantifying how successfully interactive voice response (IVR) systems satisfy callers’ goals and expectations has historically proven to be a most difficult task. Such difficulties in assessing automated customer care contact centers can be traced to two assumptions made by most stakeholders in the call center industry: 1. Performance can be effectively measured by deriving statistics from call logs; and 2. The overall performance of an IVR can be expressed by a single numeric value.

[1]  Lawrence R. Rabiner,et al.  A tutorial on hidden Markov models and selected applications in speech recognition , 1989, Proc. IEEE.

[2]  Victor Zue,et al.  Experiments in Evaluating Interactive Spoken Language Systems , 1992, HLT.

[3]  J. Ross Quinlan,et al.  C4.5: Programs for Machine Learning , 1992 .

[4]  Elizabeth Shriberg,et al.  Human-Machine Problem Solving Using Spoken Language Systems (SLS): Factors Affecting Performance and User Satisfaction , 1992, HLT.

[5]  Morena Danieli,et al.  Metrics for Evaluating Dialogue Strategies in a Spoken Language System , 1996, ArXiv.

[6]  Giuseppe Riccardi,et al.  How may I help you? , 1997, Speech Commun..

[7]  Merriam-Webster.  Merriam-Webster's Collegiate Dictionary , 1998 .

[8]  Marilyn A. Walker,et al.  From novice to expert: the effect of tutorials on user expertise with spoken dialogue systems , 1998, ICSLP.

[9]  Robert Graham,et al.  Towards a tool for the Subjective Assessment of Speech System Interfaces (SASSI) , 2000, Natural Language Engineering.

[10]  Manny Rayner,et al.  Comparing grammar-based and robust approaches to speech understanding: a case study , 2001, INTERSPEECH.

[11]  Research Issues for the Next Generation Spoken Dialogue Systems Revisited , 2001, TSD.

[12]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[13]  Maxine Eskénazi,et al.  Let's go public! taking a spoken dialog system to the real world , 2005, INTERSPEECH.

[14]  A.I. Rudnicky,et al.  Constructing accurate beliefs in spoken dialog systems , 2005, IEEE Workshop on Automatic Speech Recognition and Understanding, 2005..

[15]  Wolfgang Minker,et al.  Speech and Human-Machine Dialog , 2006 .

[16]  Roberto Pieraccini,et al.  VALUE-BASED OPTIMAL DECISION FOR DIALOG SYSTEMS , 2006, 2006 IEEE Spoken Language Technology Workshop.

[17]  Jason D. Williams,et al.  Partially Observable Markov Decision Processes for Spoken Dialogue Management , 2006 .

[18]  Roberto Pieraccini,et al.  Technical Support Dialog Systems:Issues, Problems, and Solutions , 2007, HLT-NAACL 2007.

[19]  David Suendermann-Oeft,et al.  Call classification for automated troubleshooting on large corpora , 2007, 2007 IEEE Workshop on Automatic Speech Recognition & Understanding (ASRU).

[20]  Hakan Melin,et al.  CTT-bank: A speech controlled telephone banking system - an initial evaluation , 2007 .

[21]  Hui Ye,et al.  The Hidden Information State Approach to Dialog Management , 2007, 2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP '07.

[22]  David Suendermann-Oeft,et al.  Caller Experience: A method for evaluating dialog systems and its automatic prediction , 2008, 2008 IEEE Spoken Language Technology Workshop.

[23]  Jason D. Williams.  Exploiting the ASR n-best by tracking multiple dialog state hypotheses , 2008, INTERSPEECH.

[24]  David Suendermann-Oeft,et al.  Call Classification with Hundreds of Classes and Hundred Thousands of Training Utterances ... ... and No Target Domain Data , 2008, PIT.

[25]  David Suendermann-Oeft,et al.  From rule-based to statistical grammars: Continuous improvement of large-scale spoken dialog systems , 2009, 2009 IEEE International Conference on Acoustics, Speech and Signal Processing.

[26]  David Suendermann-Oeft,et al.  A Handsome Set of Metrics to Measure Utterance Classification Performance in Spoken Dialog Systems , 2009, SIGDIAL Conference.

[27]  David Suendermann,et al.  How to drink from a fire hose: one person can annoscribe 693 thousand utterances in one month , 2010, SIGDIAL 2010.