Mean Average Distance to Resolver: An Evaluation Metric for Ticket Routing in Expert Network

In the technical support division of a large enterprise software provider, customers' technical incidents, problems, and change requests are processed as tickets. Each ticket is assigned to a support engineer for processing. Due to the limited expertise of individuals, resolving a ticket may involve routing the ticket among multiple groups of engineers. Each routing step costs time and resources. It is desirable for experts to route a ticket to its most likely resolver with minimum steps. Automated or semi-automated systems are proposed to improve routing efficiency. To evaluate the performance of any system, including human routing, two metrics are commonly used, namely Mean Steps to Resolver (MSTR) and Resolution Rate (RR). The two measures are designed independently, with different objectives and at different scales, making it difficult to compare systems. Moreover, the current measures only consider the resolver group as the ground truth, even during path-level evaluation. They disregard the contribution of intermediate groups during the ticket resolution. In this paper, we propose a distance-based unified evaluation measure named Mean Average Distance to Resolver (MADR). This new framework addresses the aforementioned limitations, and it can be easily modified to adapt to different business requirements in different organizations. In addition, existing evaluation paradigm does not consider human routing steps except the resolver. We argue that the predicted paths may not be followed exactly by expert groups in real operation. An assistive routing evaluation framework is therefore designed to take into account expert's choice when recommendation fails, for each routing. Experiments using proprietary data from a large enterprise demonstrate that MADR can be used to benchmark and compare routing systems.

[1]  Éva Tardos,et al.  Maximizing the Spread of Influence through a Social Network , 2015, Theory Comput..

[2]  Yang Li,et al.  Analyzing expert behaviors in collaborative networks , 2014, KDD.

[3]  Jimeng Sun,et al.  AIM-HI: A framework for request routing in large-scale IT global service delivery , 2009, IBM J. Res. Dev..

[4]  Louise E. Moser,et al.  Understanding task-driven information flow in collaborative networks , 2012, WWW.

[5]  Rajeev Gupta,et al.  Automating ITSM Incident Management Process , 2008, 2008 International Conference on Autonomic Computing.

[6]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[7]  Rajiv Ramnath,et al.  Probabilistic Sequence Modeling for Trustworthy IT Servicing by Collective Expert Networks , 2016, 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC).

[8]  Louise E. Moser,et al.  Generative models for ticket resolution in expert networks , 2010, KDD.

[9]  Rajiv Ramnath,et al.  Motivating dynamic features for resolution time estimation within IT operations management , 2016, 2016 IEEE International Conference on Big Data (Big Data).

[10]  Hamideh Afsarmanesh,et al.  The Emerging Discipline of Collaborative Networks , 2004, Virtual Enterprises and Collaborative Networks.

[11]  Akila Venkatesan,et al.  Novel Metrics for Bug Triage , 2014, J. Softw..

[12]  Deng Cai,et al.  Distributed Representations of Expertise , 2016, SDM.

[13]  Bikram Sengupta,et al.  SmartDispatch: enabling efficient ticket dispatch in an IT service environment , 2012, KDD.

[14]  Claudio Bartolini,et al.  Next Best Step and Expert Recommendation for Collaborative Processes in IT Service Management , 2011, BPM.

[15]  Mudhakar Srivatsa,et al.  Query Answering Efficiency in Expert Networks Under Decentralized Search , 2016, CIKM.

[16]  Louise E. Moser,et al.  Reliable Ticket Routing in Expert Networks , 2012 .

[17]  Yi Chen,et al.  Content-Aware Resolution Sequence Mining for Ticket Routing , 2010, BPM.

[18]  Yi Chen,et al.  Assessing Expertise Awareness in Resolution Networks , 2010, 2010 International Conference on Advances in Social Networks Analysis and Mining.

[19]  Massimiliano Di Penta,et al.  An approach to classify software maintenance requests , 2002, International Conference on Software Maintenance, 2002. Proceedings..

[20]  Yi Chen,et al.  Efficient ticket routing by resolution sequence mining , 2008, KDD.

[21]  Yi Chen,et al.  EasyTicket: a ticket routing recommendation engine for enterprise problem resolution , 2008, Proc. VLDB Endow..