“cba to check the spelling”: Investigating Parser Performance on Discussion Forum Posts

Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the ACL, pages 381–384, Los Angeles, California, June 2010. c⃝2010 Association for Computational Linguistics “cba to check the spelling” Investigating Parser Performance on Discussion Forum Posts Jennifer Foster National Centre for Language Technology School of Computing Dublin City University jfoster@computing.dcu.ie Abstract We evaluate the Berkeley parser on text from an online discussion forum. We evaluate the parser output with and without gold tokens and spellings (using Sparseval and Parseval), and we compile a list of problematic phenom- ena for this domain. The Parseval f-score for a small development set is 77.56. This increases to 80.27 when we apply a set of simple trans- formations to the input sentences and to the Wall Street Journal (WSJ) training sections. 1 Introduction Parsing techniques have recently become efficient enough for parsers to be used as part of a pipeline in a variety of tasks. Another recent development is the rise of user-generated content in the form of blogs, wikis and discussion forums. Thus, it is both inter- esting and necessary to investigate the performance of NLP tools trained on edited text when applied to unedited Web 2.0 text. McClosky et al. (2006) re- port a Parseval f-score decrease of 5% when a WSJ- trained parser is applied to Brown corpus sentences. In this paper, we move even further from the WSJ by investigating the performance of the Berkeley parser (Petrov et al., 2006) on user-generated content. We create gold standard phrase structure trees for the posts on two threads of the same online dis- cussion forum. We then parse the sentences in one thread, our development set, with the Berke- ley parser under three conditions: 1) when it per- forms its own tokenisation, 2) when it is provided with gold tokens and 3) when misspellings in the in- put have been corrected. A qualitative evaluation is then carried out on parser output under the third con- dition. Based on this evaluation, we identify some “low-hanging fruit” which we attempt to handle ei- ther by transforming the input sentence or by trans- forming the WSJ training material. The success of these transformations is evaluated on our develop- ment and test sets, with encouraging results. 2 Parser Evaluation Experiments Data Our data consists of sentences that occur on the BBC Sport 606 Football discussion forum. The posts on this forum are quite varied, ranging from throwaway comments to more considered essay-like contributions. The development set consists of 42 posts (185 sentences) on a thread discussing a con- troversial refereeing decision in a soccer match.1 The test set is made up of 40 posts (170 sentences) on a thread discussing a player’s behaviour in the same match.2 The average sentence length in the development set is 18 words and the test set 15 words. Tokenisation and spelling correction were carried out by hand on the sentences in both sets.3 They were then parsed using Bikel’s parser (Bikel, 2004) and corrected by hand using …

Content truncated to 500 words