Corpora of Typical Sentences

Corpora of Typical Sentences Lydia M¨uller, Uwe Quasthoff, Maciej Sumalvico University of Leipzig Augustusplatz 10, 04109 Leipzig {lydia,quasthoff,sumalvico}@informatik.uni-leipzig.de Abstract Typical sentences of characteristic syntactic structures can be used for language understanding tasks like finding typical slotfiller for verbs. The paper describes the selection of such typical sentences representing usually about 5% of the original corpus. The sentences are selected by the frequency of the corresponding POS tag sequence together with an entropy theshold, and the selection method is shown to work language independently. Entropy measuring the distribution of words in a given position turns out to identify larger sets of near-duplicate sentences, not considered typical. A statistical comparison of those subcorpora with the underlying corpus shows the intended shorter sentence length, but also a decrease of word frequencies for function words associated to more complex sentences. Keywords: typical sentences, sentence signatures, language statistics, corpus comparison 1. Introduction Statistical analyses of language are usually based on large corpora compiled from publicly available written sources, e.g. news, Wikipedia, crawled webpages or literature (Ba- roni and Bernardini 2004, Biemann et al. 2013). Compared to everyday speech, such sources tend to be biased towards long sentences and complex syntactic structures. There have been attempts to compile corpora more represantative of everyday language by utilizing different sources, espe- cially movie subtitles (Lison and Tiedemann 2016). How- ever, the availability of such sources is limited. In this paper, we propose a method for diminishing the bias towards complex syntactic structures usually found in large corpora. In order to accomplish this, we select “typical sentences”, defined as sentences with a common syntactic structure (represented as a sequence of POS-tags). In con- trast to simplified or controlled languages (like Simplified English (Ogden 1932) or Kontrolliertes Deutsch (Lehrndor- fer 1996) (controlled German) there is no set of handwritten rules for syntax and vocabulary. However, the sublanguage emerging in typical sentences may be considered as an au- tomatically genarated analogy. As an illustration of our approach, we currently provide typical sentence corpora for English, German, French, Dutch and Italian as a part of the Leipzig Corpora Collec- tion.1 Random samples of up to 1 million sentences are freely available for download. 2. Selecting typical sentences Given a POS-tagged sentence, we define the sentence sig- nature as the corresponding sequence of POS tags. For a corpus of sentences, such signatures can be ordered by fre- quency. As one could expect, the most frequent sentence signatures belong to relatively short sentences with typical structure. Table 1 shows the top five sentence signatures from newspaper corpora of the Leipzig Corpora Collection (Goldhahn et al. 2012) with sample sentences for English. As POS tagger, the Stuttgart Tree Tagger (Schmid 1994) is used. 1http://corpora.uni-leipzig.de The following two properties of the list of sentence signa- tures are not so obvious: There are frequent sentence sig- natures belonging to large sets of near-duplicate sentences. They are usually unexpectedly long and contain numbers. The most frequent examples are given in Table 2. Such sentences are not considered typical …

Content truncated to 500 words