From News to Comment: Resources and Benchmarks for Parsing the Language of Web 2.0

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 893–901, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP From News to Comment: Resources and Benchmarks for Parsing the Language of Web 2.0 Jennifer Foster1, ¨Ozlem C¸ etino˘glu1, Joachim Wagner1, Joseph Le Roux2 Joakim Nivre3, Deirdre Hogan1 and Josef van Genabith1 1,3NCLT/CNGL, Dublin City University, Ireland 2LIF - CNRS UMR 6166, Universit´e Aix-Marseille, France 3Department of Linguistics and Philology, Uppsala University, Sweden 1{jfoster,ocetinoglu,jwagner,dhogan,josef}@computing.dcu.ie 2joseph.le-roux@lif.univ-mrs.fr, 3joakim.nivre@lingfil.uu.se Abstract We investigate the problem of parsing the noisy language of social media. We evalu- ate four Wall-Street-Journal-trained statis- tical parsers (Berkeley, Brown, Malt and MST) on a new dataset containing 1,000 phrase structure trees for sentences from microblogs (tweets) and discussion forum posts. We compare the four parsers on their ability to produce Stanford depen- dencies for these Web 2.0 sentences. We find that the parsers have a particular prob- lem with tweets and that a substantial part of this problem is related to POS tagging accuracy. We attempt three retraining ex- periments involving Malt, Brown and an in-house Berkeley-style parser and obtain a statistically significant improvement for all three parsers. 1 Introduction With the explosive growth in social media, nat- ural language processing technologies, including parsers, need to adapt to reflect the linguistic changes brought about by new forms of online communication. The availability of the Penn Tree- bank has encouraged much research in supervised parsing for English and facilitated comparision be- tween parsers. This has led to impressive perfor- mance for in-domain parsing. Some progress has also been achieved in adapting parsers to new do- mains using semi-supervised and unsupervised ap- proaches involving some labelled source domain training data, little, if any, labelled target domain data and large quantities of unlabelled target do- main data. Much of the work on parser adapta- tion has focused on biomedical text and questions - very little has focused on the informal language prevalent in much of the user-generated content of Web 2.0. Domain adaptation to the language of social media is particularly challenging since Web 2.0 is not really a domain, consisting, as it does, of utterances from a wide variety of speakers from different geographical and social backgrounds. Foster (2010) carried out a pilot study on this topic by investigating the performance of the Berkeley parser (Petrov et al., 2006) on sentences taken from a sports discussion forum. Each mis- parsed sentence was examined manually and a list of problematic phenomena identified. We extend this work by looking at a larger dataset consisting not only of discussion forum posts but also microblogs or tweets. We extend the parser evaluation to the Brown reranking parser (Charniak and Johnson, 2005), MaltParser (Nivre et al., 2006) and MSTParser (McDonald et al., 2005), and we examine the ability of all four parsers to recover typed Stanford dependencies (de Marneffe et al., 2006). The relative ranking of the four parsers confirms the results of pre- vious Stanford-dependency-based parser evalua- tions on other datasets (Cer et …

Content truncated to 500 words