Foreebank: Syntactic Analysis of Customer Support Forums

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1341–1347, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Foreebank: Syntactic Analysis of Customer Support Forums Rasoul Kaljahi1, Jennifer Foster1 Johann Roturier2 Corentin Ribeyre3, Teresa Lynn1, Joseph Le Roux4 1ADAPT Centre, School of Computing, Dublin City University, Ireland {rkaljahi, jfoster, tlynn}@computing.dcu.ie 2Symantec Research Labs, Dublin, Ireland johann roturier@symantec.com 3Alpage, INRIA, Univ Paris Diderot, Sorbonne Paris Cit´e, France corentin.ribeyre@inria.fr 4Universit´e Paris Nord, France joseph.le.roux@gmail.com Abstract We present a new treebank of English and French technical forum content which has been annotated for grammatical errors and phrase structure. This double annotation allows us to empirically measure the effect of errors on parsing performance. While it is slightly easier to parse the corrected versions of the forum sentences, the errors are not the main factor in making this kind of text hard to parse. 1 Introduction The last five years has seen a considerable amount of research carried out on web and social me- dia text parsing, with new treebanks being cre- ated (Foster et al., 2011; Seddah et al., 2012; Mott et al., 2012; Kong et al., 2014), and new parsing systems developed (Petrov and McDonald, 2012; Kong et al., 2014). In this paper we explore a par- ticular source of user-generated text, namely, posts from technical support forums, which are a pop- ular means for customers to resolve their queries about a product. An accurate parser for this kind of text can be used to inform forum-level question- answering, machine translation and quality esti- mation of machine translation. We create a 2000-sentence treebank called Foreebank which contains sentences from the Symantec Norton English and French technical support forums.1 The phrase structure of the sen- tences is annotated and any grammatical errors are marked in the trees. Marking the grammatical er- rors allows us to precisely measure the amount of grammatical noise in this kind of text, and joint er- ror and syntactic annotation allows us to determine its effect on parsing. 1http://community.norton.com Foster (2010) explored the effect of spelling er- rors on parsing performance of conversational fo- rum text. We extend this study to include gram- matical errors, focusing on more technical con- tent. Foster et al. (2008) explored the effect of artificially generated grammatical errors on Wall Street Journal parsing. We concentrate on forum text rather than newspaper text, and, crucially, ex- amine the effect of real grammatical errors. We find that the level of grammatical noise is lower than expected, with capitalisation and punctuation errors being the most frequent. While correcting all the errors does result in a performance increase of 1.5% for English and 0.8% for French, the ma- jor challenge in parsing these sentences seems not to be “bad language” (Eisenstein, 2013) per se. The main contribution of the paper is the Foree- bank data set itself2 but we also carry out prelim- inary parsing experiments evaluating the accuracy of a PCFG-LA parser on Foreebank, examining the effect of grammatical errors on …

Content truncated to 500 words