Improving meeting summarization by focusing on user needs: a task-oriented evaluation

Advances in multimedia technologies have enabled the creation of huge archives of audio-video recordings of meetings, and there is burgeoning interest in developing meeting browsers to help users better leverage these archives. A recent study has shown that extractive summaries provide a more efficient way of navigating meeting content than simply reading through the transcript and using the audio-video record, or navigating via keyword search (Murray, 2007). The extractive summary technique identifies informative dialogue acts to generate general purpose summaries. These summaries can still be lengthy. Recently, we have developed a decision-focused summarization system that presents only 1-2% of the recordings related to decision making. In this paper, we describe a task-based evaluation in which we compare the decision-focused summaries to the general purpose summaries. Our results indicate that the more focused summaries help users perform the decision debriefing task more effectively and improve perceived efficiency. In addition, this study also investigates the effect of automatic summaries and transcription on task effectiveness, report quality, and users' perceptions of task success.

[1]  H. P. Edmundson,et al.  New Methods in Automatic Extracting , 1969, JACM.

[2]  Francine Chen,et al.  A trainable document summarizer , 1995, SIGIR '95.

[3]  Inderjeet Mani,et al.  Machine Learning of Generic and User-Focused Summarization , 1998, AAAI/IAAI.

[4]  Jay F. Nunamaker,et al.  Meeting analysis: findings from research and practice , 2001, Proceedings of the 34th Annual Hawaii International Conference on System Sciences.

[5]  Konstantinos Koumpis,et al.  Extractive summarization of voicemail using lexical and prosodic feature subset selection , 2001, INTERSPEECH.

[6]  Marc Moens,et al.  Articles Summarizing Scientific Articles: Experiments with Relevance and Rhetorical Status , 2002, CL.

[7]  Klaus Zechner,et al.  Automatic Summarization of Open-Domain Multiparty Dialogues in Diverse Genres , 2002, CL.

[8]  Kathleen R. McKeown,et al.  Information fusion for multidocument summarization: paraphrasing and generation , 2003 .

[9]  Wilfried Post,et al.  A research environment for meeting behavior , 2004 .

[10]  Mike Flynn,et al.  Browsing Recorded Meetings with Ferret , 2004, MLMI.

[11]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[12]  Andrei Popescu-Belis,et al.  User Query Analysis for the Specification and Evaluation of a Dialogue Processing and Retrieval System , 2004, LREC.

[13]  Dirk Heylen,et al.  Argument Diagramming of Meeting Conversations , 2005 .

[14]  V. Pallotta.  Collaborative and Argumentative Models of Meeting Discussions , 2005 .

[15]  Roeland Ordelman,et al.  Transcription of conference room meetings: an investigation , 2005, INTERSPEECH.

[16]  Carolyn Penstein Rosé,et al.  The Necessity of a Meeting Recording and Playback System, and the Benefit of Topic-Level Annotations to Meeting Browsing , 2005, INTERACT.

[17]  Steve Whittaker,et al.  Analysing Meeting Records: An Ethnographic Study and Technological Implications , 2005, MLMI.

[18]  Jean Carletta,et al.  Extractive summarization of meeting recordings , 2005, INTERSPEECH.

[19]  Julia Hirschberg,et al.  Comparing lexical, acoustic/prosodic, structural and discourse features for speech summarization , 2005, INTERSPEECH.

[20]  Jean Carletta,et al.  The AMI Meeting Corpus: A Pre-announcement , 2005, MLMI.

[21]  Johanna D. Moore,et al.  What Decisions Have You Made?: Automatic Decision Detection in Meeting Conversations , 2007, HLT-NAACL.

[22]  Violeta Seretan,et al.  User Requirements Analysis for Meeting Information Retrieval Based on Query Elicitation , 2007, ACL.

[23]  Michael Strube,et al.  Improving extractive dialogue summarization by utilizing human feedback , 2007, Artificial Intelligence and Applications.