An Improved Phrase-based Approach to Annotating and Summarizing Student Course Responses

Teaching large classes remains a great challenge, primarily because it is difficult to attend to all the student needs in a timely manner. Automatic text summarization systems can be leveraged to summarize the student feedback, submitted immediately after each lecture, but it is left to be discovered what makes a good summary for student responses. In this work we explore a new methodology that effectively extracts summary phrases from the student responses. Each phrase is tagged with the number of students who raise the issue. The phrases are evaluated along two dimensions: with respect to text content, they should be informative and well-formed, measured by the ROUGE metric; additionally, they shall attend to the most pressing student needs, measured by a newly proposed metric. This work is enabled by a phrase-based annotation and highlighting scheme, which is new to the summarization task. The phrase-based framework allows us to summarize the student responses into a set of bullet points and present to the instructor promptly.

[1]  Diane J. Litman,et al.  Summarizing Student Responses to Reflection Prompts , 2015, EMNLP.

[2]  Giuseppe Carenini,et al.  Abstractive Meeting Summarization with Entailment and Fusion , 2013, ENLG.

[3]  Koji Yatani,et al.  Review spotlight: a user interface for summarizing user-generated reviews using adjective-noun word pairs , 2011, CHI.

[4]  T. Landauer,et al.  Indexing by Latent Semantic Analysis , 1990 .

[5]  Vasile Rus,et al.  SEMILAR: The Semantic Similarity Toolkit , 2013, ACL.

[6]  Stephen Krause,et al.  The effectiveness of students' daily reflections on learning in engineering context , 2011 .

[7]  Yoshihiro Ueda,et al.  Toward the "At-a-glance" Summary: Phrase-representation Summarization Method , 2000, COLING.

[8]  Sumit Basu,et al.  Powergrading: a Clustering Approach to Amplify Human Effort for Short Answer Grading , 2013, TACL.

[9]  Lu Wang,et al.  Neural Network-Based Abstract Generation for Opinions and Arguments , 2016, NAACL.

[10]  Bowen Zhou,et al.  Sequence-to-Sequence RNNs for Text Summarization , 2016, ArXiv.

[11]  Michalis Vazirgiannis,et al.  Clustering and Community Detection in Directed Networks: A Survey , 2013, ArXiv.

[12]  Hoa Trang Dang,et al.  Overview of the TAC 2008 Update Summarization Task , 2008, TAC.

[13]  Omer Levy,et al.  word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding method , 2014, ArXiv.

[14]  Dragomir R. Radev,et al.  LexRank: Graph-based Lexical Centrality as Salience in Text Summarization , 2004, J. Artif. Intell. Res..

[15]  George A. Miller,et al.  WordNet: A Lexical Database for English , 1995, HLT.

[16]  Jason Weston,et al.  A Neural Attention Model for Abstractive Sentence Summarization , 2015, EMNLP.

[17]  David Jurgens,et al.  Word Sense Induction by Community Detection , 2011, Graph-based Methods for Natural Language Processing.

[18]  Peter J. Rousseeuw,et al.  Clustering by means of medoids , 1987 .

[19]  Min-Yen Kan.  Keywords, phrases, clauses and sentences: topicality, indicativeness and informativeness at scales , 2015, ACL 2015.

[20]  Fred Paas,et al.  Reflection prompts and tutor feedback in a web-based learning environment: effects on students' self-regulated learning competence , 2004, Comput. Hum. Behav..

[21]  Dragomir R. Radev,et al.  Learning From Collective Human Behavior to Introduce Diversity in Lexical Choice , 2011, ACL.

[22]  Zhiyuan Liu,et al.  Clustering to Find Exemplar Terms for Keyphrase Extraction , 2009, EMNLP.

[23]  Andrew McCallum,et al.  Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data , 2001, ICML.

[24]  Diane J. Litman,et al.  Empirical analysis of exploiting review helpfulness for extractive summarization of online reviews , 2014, COLING.

[25]  Dan Klein,et al.  Jointly Learning to Extract and Compress , 2011, ACL.

[26]  Shibamouli Lahiri,et al.  Building a Dataset for Summarization and Keyword Extraction from Emails , 2014, LREC.

[27]  Jingtao Wang,et al.  Enhancing Instructor-Student and Student-Student Interactions with Mobile Interfaces and Summarization , 2015, NAACL.

[28]  Chin-Yew Lin,et al.  ROUGE: A Package for Automatic Evaluation of Summaries , 2004, ACL 2004.

[29]  Vincent Ng,et al.  Automatic Keyphrase Extraction: A Survey of the State of the Art , 2014, ACL.

[30]  Noah A. Smith,et al.  Summarization with a Joint Model for Sentence Extraction and Compression , 2009, ILP 2009.

[31]  William M. Campbell,et al.  Content+Context=Classification: Examining the Roles of Social Interactions and Linguist Content in Twitter User Classification , 2014, SocialNLP@COLING.

[32]  Ani Nenkova,et al.  Evaluating Content Selection in Summarization: The Pyramid Method , 2004, NAACL.

[33]  Dan Klein,et al.  Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints , 2016, ACL.

[34]  Ahmet Aker,et al.  The SENSEI Annotated Corpus: Human Summaries of Reader Comment Conversations in On-line News , 2016, SIGDIAL Conference.

[35]  Salim Roukos,et al.  Bleu: a Method for Automatic Evaluation of Machine Translation , 2002, ACL.

[36]  Vasile Rus,et al.  Latent Semantic Analysis Models on Wikipedia and TASA , 2014, LREC.

[37]  Yi-fang Brook Wu,et al.  Domain-specific keyphrase extraction , 2005, CIKM '05.

[38]  Frederick Mosteller,et al.  The "Muddiest Point in the Lecture" as a feedback device , 1989 .

[39]  Diane J. Litman,et al.  Determining the Quality of a Student Reflective Response , 2016, FLAIRS Conference.

[40]  Fei Liu,et al.  Document Summarization via Guided Sentence Compression , 2013, EMNLP.

[41]  Fei Liu,et al.  Automatic Summarization of Student Course Feedback , 2016, HLT-NAACL.

[42]  Jingtao Wang,et al.  CourseMIRROR: Enhancing Large Classroom Instructor-Student Interactions via Mobile Interfaces and Natural Language Processing , 2015, CHI Extended Abstracts.

[43]  Ian H. Witten,et al.  Human-competitive tagging using automatic keyphrase extraction , 2009, EMNLP.

[44]  Noah A. Smith,et al.  Toward Abstractive Summarization Using Semantic Representations , 2018, NAACL.

[45]  François Yvon,et al.  Practical Very Large Scale CRFs , 2010, ACL.

[46]  D. Boud,et al.  Reflection, turning experience into learning , 1985 .

[47]  Santo Fortunato,et al.  Finding Statistically Significant Communities in Networks , 2010, PloS one.

[48]  Using Word Clouds for Fast, Formative Assessment of Students’ Short Written Responses , 2014 .