Towards Robust Linguistic Analysis using OntoNotes

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143–152, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Towards Robust Linguistic Analysis Using OntoNotes Sameer Pradhan1, Alessandro Moschitti2,3, Nianwen Xue4, Hwee Tou Ng5 Anders Bj¨orkelund6, Olga Uryupina2, Yuchen Zhang4 and Zhi Zhong5 1 Boston Childrens Hospital and Harvard Medical School, Boston, MA 02115, USA 2 University of Trento, University of Trento, 38123 Povo (TN), Italy 3 QCRI, Qatar Foundation, 5825 Doha, Qatar 4 Brandeis University, Brandeis University, Waltham, MA 02453, USA 5 National University of Singapore, Singapore, 117417 6 University of Stuttgart, 70174 Stuttgart, Germany Abstract Large-scale linguistically annotated cor- pora have played a crucial role in advanc- ing the state of the art of key natural lan- guage technologies such as syntactic, se- mantic and discourse analyzers, and they serve as training data as well as evaluation benchmarks. Up till now, however, most of the evaluation has been done on mono- lithic corpora such as the Penn Treebank, the Proposition Bank. As a result, it is still unclear how the state-of-the-art analyzers perform in general on data from a vari- ety of genres or domains. The completion of the OntoNotes corpus, a large-scale, multi-genre, multilingual corpus manually annotated with syntactic, semantic and discourse information, makes it possible to perform such an evaluation. This paper presents an analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus. This should set the bench- mark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance. 1 Introduction Roughly a million words of text from the Wall Street Journal newswire (WSJ), circa 1989, has had a significant impact on research in the lan- guage processing community — especially those in the area of syntax and (shallow) semantics, the reason for this being the seminal impact of the Penn Treebank project which first selected this text for annotation. Taking advantage of a solid syn- tactic foundation, later researchers who wanted to annotate semantic phenomena on a relatively large scale, also used it as the basis of their annota- tion. For example the Proposition Bank (Palmer et al., 2005), BBN Name Entity and Pronoun coref- erence corpus (Weischedel and Brunstein, 2005), the Penn Discourse Treebank (Prasad et al., 2008), and many other annotation projects, all annotate the same underlying body of text. It was also con- verted to dependency structures and other syntac- tic formalisms such as CCG (Hockenmaier and Steedman, 2002) and LTAG (Shen et al., 2008), thereby creating an even bigger impact through these additional syntactic resources. The most re- cent one of these efforts is the OntoNotes corpus (Weischedel et al., 2011). However, unlike the previous extensions of the Treebank, in addition to using roughly a third of the same WSJ subcor- pus, OntoNotes also added several other genres, and covers two other languages — Chinese and Arabic: …

Content truncated to 500 words