Sentence Level Human Translation Quality Estimation with Attention-based Neural Networks

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 172–181 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 172 Orthographic Codes and the Neighborhood Effect: Lessons from Information Theory St´ephan Tulkens, Dominiek Sandra, Walter Daelemans CLiPS - Computational Linguistics Group Department of Linguistics University of Antwerp {firstname.lastname}@uantwerpen.be Abstract We consider the orthographic neighborhood effect: the effect that words with more orthographic similarity to other words are read faster. The neighborhood effect serves as an important control variable in psycholinguistic studies of word reading, and explains variance in addition to word length and word frequency. Following previous work, we model the neighborhood effect as the average distance to neighbors in feature space for three feature sets: slots, character ngrams and skipgrams. We optimize each of these feature sets and find evidence for language-independent optima, across five megastudy corpora from five alphabetic languages. Additionally, we show that weighting features using the inverse of mutual information (MI) improves the neighborhood effect significantly for all languages. We analyze the inverse feature weighting, and show that, across languages, grammatical morphemes get the lowest weights. Finally, we perform the same experiments on Korean Hangul, a non-alphabetic writing system, where we find the opposite results: slower responses as a function of denser neighborhoods, and a negative effect of inverse feature weighting. This raises the question of whether this is a cognitive effect, or an effect of the way we represent Hangul orthography, and indicates more research is needed. Keywords: Cognitive Methods, Lexical Database 1. Introduction One of the core issues in contemporary models of word reading is the representation of orthography. Orthography, in this case, can be understood as the visual information of a word as it is represented by the brain. It therefore does not generally refer to the actual visual presentation of a word, such as the font and case, but to a more abstract visual rep- resentation (Dehaene et al., 2005; Dehaene, 2009). One of the reasons orthography is such a core issue is that older models, such as the Interactive Activation (McClelland and Rumelhart, 1981) and DISLEX (Miikkulainen, 1993; Mi- ikkulainen, 1997) models, assumed that words were repre- sented as an array of slots, an assumption that has recently been shown to be false. As readers are remarkably flexible at decoding orthographic information from jumbled strings (Schoonbaert and Grainger, 2004; Perea et al., 2008), the positional information in orthographic representations has to be flexible to some degree; if this were not the case, readers would not be able to decode transposition neigh- bors, e.g., JUGDE - JUDGE, efficiently. These, and other, phenomena have motivated the search for various models or featurizations of orthography that can accurately capture empirical data, which has also been dubbed the search for an orthographic code (Grainger, 2008; Grainger, 2018). Ex- amples of such feature sets, or orthographic codes, are char- acter ngrams, or wickelgraphs (Wickelgren, 1969), four- teen segment coding (Rumelhart and Siple, 1974), slot- based coding (McClelland and …

Content truncated to 500 words