Unsupervised Induction of Linguistic Categories with Records of Reading, Speaking, and Writing

Proceedings of NAACL-HLT 2018, pages 2028–2038 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Unsupervised Induction of Linguistic Categories with Records of Reading, Speaking, and Writing Maria Barrett1 Ana V. Gonz´alez-Gardu˜no2 Lea Frermann3 Anders Søgaard2 1Department of Nordic Studies and Linguistics, University of Copenhagen, Denmark 2Department of Computer Science, University of Copenhagen, Denmark 3Amazon Development Center, Berlin, Germany barrett@hum.ku.dk {ana, soegaard}@di.ku.dk lfrerman@amazon.com Abstract When learning POS taggers and syntactic chunkers for low-resource languages, differ- ent resources may be available, and often all we have is a small tag dictionary, mo- tivating type-constrained unsupervised induc- tion. Even small dictionaries can improve the performance of unsupervised induction algo- rithms. This paper shows that performance can be further improved by including data that is readily available or can be easily obtained for most languages, i.e., eye-tracking, speech, or keystroke logs (or any combination thereof). We project information from all these data sources into shared spaces, in which the union of words is represented. For English unsuper- vised POS induction, the additional informa- tion, which is not required at test time, leads to an average error reduction on Ontonotes do- mains of 1.5% over systems augmented with state-of-the-art word embeddings. On Penn Treebank the best model achieves 5.4% error reduction over a word embeddings baseline. We also achieve significant improvements for syntactic chunk induction. Our analysis shows that improvements are even bigger when the available tag dictionaries are smaller. 1 Introduction It is a core assumption in linguistics that humans have knowledge of grammar and that they use this knowledge to generate and process language. Reading, writing, and talking leave traces of this knowledge and in psycholinguistics this data is used to analyze our grammatical competencies. Psycholinguists are typically interested in falsify- ing a specific hypothesis about our grammatical competencies and therefore collect data with this hypothesis in mind. In NLP, we typically require big, representative corpora. NLP usually has in- Lea Frermann carried out this work while at the Univer- sity of Edinburgh. duced the models from expensive corpus annota- tions by professional linguists, but recently, a few researchers have shown that data traces from hu- man processing can be used directly to improve NLP models (Klerke et al., 2016; Barrett et al., 2016; Plank, 2016). In this paper, we investigate whether unsuper- vised POS induction and unsupervised syntactic chunking can be improved using human text pro- cessing traces. We also explore what traces are beneficial, and how they are best combined. Our work supplements psycholinguistic research by evaluating human data on larger scale than usual, but more robust unsupervised POS induction also contributes to NLP for low-resource languages for which professional annotators are hard to find, and where instead, data from native speakers can be used to augment unsupervised learning. We explore three different modalities of data reflecting human processing plus standard, pre- trained distributional word embeddings for com- parison, but also because some modalities might fare better when combined with distributional vectors. Data reflecting human processing …

Content truncated to 500 words