SemEval-2016 Task 8: Meaning Representation Parsing

In this report we summarize the results of the SemEval 2016 Task 8: Meaning Representation Parsing. Participants were asked to generate Abstract Meaning Representation (AMR) (Banarescu et al., 2013) graphs for a set of English sentences in the news and discussion forum domains. Eleven sites submitted valid systems. The availability of state-of-the-art baseline systems was a key factor in lowering the bar to entry; many submissions relied on CAMR (Wang et al., 2015b; Wang et al., 2015a) as a baseline system and added extensions to it to improve scores. The evaluation set was quite difficult to parse, particularly due to creative approaches to word representation in the web forum portion. The top scoring systems scored 0.62 F1 according to the Smatch (Cai and Knight, 2013) evaluation heuristic. We show some sample sentences along with a comparison of system parses and perform quantitative ablative studies.

[1]  Alastair Butler.  DynamicPower at SemEval-2016 Task 8: Processing syntactic parse trees with a Dynamic Semantics core , 2016, SemEval@NAACL-HLT.

[2]  Guntis Barzdins,et al.  RIGA at SemEval-2016 Task 8: Impact of Smatch Extensions and Character-Level Neural Translation on AMR Parsing Accuracy , 2016, *SEMEVAL.

[3]  Maria Leonor Pacheco,et al.  of the Association for Computational Linguistics: , 2001 .

[4]  Yevgeniy Puzikov,et al.  M2L at SemEval-2016 Task 8: AMR Parsing with Neural Networks , 2016, SemEval@NAACL-HLT.

[5]  Jaime G. Carbonell,et al.  CMU at SemEval-2016 Task 8: Graph-based AMR Parsing with Infinite Ramp Loss , 2016, *SEMEVAL.

[6]  Daniel Marcu,et al.  Parsing English into Abstract Meaning Representation Using Syntax-Based Machine Translation , 2015, EMNLP.

[7]  Yannick Versley,et al.  ICL-HD at SemEval-2016 Task 8: Meaning Representation Parsing - Augmenting AMR Parsing with a Preposition Semantic Role Labeling Neural Network , 2016, SemEval@NAACL-HLT.

[8]  James H. Martin,et al.  CU-NLP at SemEval-2016 Task 8: AMR Parsing using LSTM-based Recurrent Neural Networks , 2016, SemEval@NAACL-HLT.

[9]  Philipp Koehn,et al.  Abstract Meaning Representation for Sembanking , 2013, LAW@ACL.

[10]  Philip Resnik,et al.  CLIP$@$UMD at SemEval-2016 Task 8: Parser for Abstract Meaning Representation using Learning to Search , 2016, SemEval@NAACL-HLT.

[11]  Kevin Knight,et al.  Smatch: an Evaluation Metric for Semantic Feature Structures , 2013, ACL.

[12]  Johan Bos,et al.  Linguistically Motivated Large-Scale NLP with C&C and Boxer , 2007, ACL.

[13]  Chuan Wang,et al.  A Transition-based Algorithm for AMR Parsing , 2015, NAACL.

[14]  Johan Bos,et al.  The Meaning Factory at SemEval-2016 Task 8: Producing AMRs with Boxer , 2016, SemEval@NAACL-HLT.

[15]  Christopher D. Manning,et al.  Robust Subgraph Generation Improves Abstract Meaning Representation Parsing , 2015, ACL.

[16]  Yang Gao,et al.  Aligning English Strings with Abstract Meaning Representation Graphs , 2014, EMNLP.

[17]  Martha Palmer,et al.  From TreeBank to PropBank , 2002, LREC.

[18]  Luke S. Zettlemoyer,et al.  Broad-coverage CCG Semantic Parsing with AMR , 2015, EMNLP.

[19]  Heng Ji,et al.  CAMR at SemEval-2016 Task 8: An Extended Transition-based AMR Parser , 2016, SemEval@NAACL-HLT.

[20]  Xiaochang Peng,et al.  UofR at SemEval-2016 Task 8: Learning Synchronous Hyperedge Replacement Grammar for AMR Parsing , 2016, *SEMEVAL.

[21]  Andreas Vlachos,et al.  UCL+Sheffield at SemEval-2016 Task 8: Imitation learning for AMR parsing with an alpha-bound , 2016, SemEval@NAACL-HLT.

[22]  Chuan Wang,et al.  Boosting Transition-based AMR Parsing with Refined Actions and Auxiliary Analyzers , 2015, ACL.

[23]  Jaime G. Carbonell,et al.  A Discriminative Graph-Based Parser for the Abstract Meaning Representation , 2014, ACL.