Biber Redux: Reconsidering Dimensions of Variation in American English

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 565–576, Dublin, Ireland, August 23-29 2014. Biber Redux: Reconsidering Dimensions of Variation in American English Rebecca J. Passonneau Center for Computational Learning Systems Columbia University New York, New York USA becky@ccls.columbia.edu Nancy Ide Department of Computer Science Vassar College Poughkeepsie, New York USA ide@cs.vassar.edu Songqiao Su Department of Computer Science Columbia University New York, New York USA ss4555@columbia.edu Jesse Stuart Department of Computer Science Vassar College Poughkeepsie, New York USA jestuart@cs.vassar.edu Abstract Genre classification has been found to improve performance in many applications of statistical NLP, including language modeling for spoken language, domain adaptation of statistical parsers, and machine translation. It has also been found to benefit retrieval of spoken or written docu- ments. At its base, however, classification assumes separability. This paper revisits an assump- tion that genre variation is continuous along multiple dimensions, and an early use of principal component analysis to find these dimensions. Results on a very heterogeneous corpus of post- 1990s American English reveal four major dimensions, three of which echo those found in prior work and the fourth depending on features not used in the earlier study. The resulting model can provide a basis for more detailed analysis of sub-genres and the relation between genre and situations of language use, as well as a means to predict distributional properties of new genres. 1 Introduction Although a precise definition of the term “genre” has traditionally proven to be elusive, it cannot be dis- puted that a genre represents a set of shared regularities among written or spoken documents that enables readers, writers, listeners and speakers to signal discourse function, and that conditions their expectations of linguistic form. Genre distinctions are therefore an important aspect of language use and understand- ing. They clearly have a role to play in statistical language processing, which relies on regularities of form as well as content. Indeed, with the advent of the Web, statistical methods for genre differenti- ation have been applied to information retrieval to limit search criteria and organize results (Karlgren and Cutting, 1994; Kessler et al., 1997; Mehler et al., 2010; Ward and Werner, 2013), and the study of genres on the web has become a sub-field in its own right (see for example (Mehler et al., 2010)). More recently, the development of genre-dependent models for a variety of natural language processing (NLP) tasks such as parsing (Ravi et al., 2008; McClosky et al., 2010; Roux et al., 2012), speech recognition (Iyer and Ostendorf, 1999), word sense disambiguation (Martinez and Agirre, 2000), and machine trans- lation (Wang et al., 2012) has been found to significantly improve performance. The ability to match documents by genre has also become important for collecting data to train language models for spoken language understanding, given the difficulty of creating large repositories of transcribed spoken language corpora (Bulyko and Ostendorf, 2003; Sarikaya et al., 2005). This work is licensed under a Creative Commons Attribution 4.0 International Licence. Page numbers …

Content truncated to 500 words