CxLM: A Construction and Context-aware Language Model

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6361–6369 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6361 CxLM: A Construction and Context-aware Language Model Yu-Hsiang Tseng, Cing-Fang Shih, Pin-Er Chen, Hsin-Yu Chou, Mao-Chang Ku, Shu-Kai Hsieh Graduate Institute of Linguistics, National Taiwan University Taipei, Taiwan seantyh@gmail.com, r08142004@ntu.edu.tw, cckk2913@gmail.com, r10142008@ntu.edu.tw, d08142002@ntu.edu.tw, shukaihsieh@ntu.edu.tw Abstract Constructions are direct form-meaning pairs with possible schematic slots. These slots are simultaneously constrained by the embedded construction itself and the sentential context. We propose that the constraint could be described by a conditional probability distribution. However, as this conditional probability is inevitably complex, we utilize language models to capture this distribution. Therefore, we build CxLM, a deep learning-based masked language model explicitly tuned to constructions’ schematic slots. We first compile a construction dataset consisting of over ten thousand constructions in Taiwan Mandarin. Next, an experiment is conducted on the dataset to examine to what extent a pretrained masked language model is aware of the constructions. We then fine-tune the model specifically to perform a cloze task on the opening slots. We find that the fine-tuned model predicts masked slots more accurately than baselines and generates both structurally and semantically plausible word samples. Finally, we release CxLM and its dataset as publicly available resources and hope to serve as new quantitative tools in studying construction grammar. Keywords: constructions, language model, slots, masked language model, construction grammar 1. Introduction Constructions are direct form-meaning pairs with- out intermediate structures. Morphemes, words, idioms, and phrasal patterns are all constructions, and there are no clear-cut boundaries between lex- icon and syntax (Fillmore, 1988). Following this approach, one does not have to assume a domain- specific cognitive process that is adapted to lan- guage (Croft, 2001). Constructions, just like sylla- bles and words, are emerged from language users’ everyday linguistic experiences: by sorting and matching incoming utterances by their similarities (Bybee, 2006). That is, the constructions do not have to be exactly the same; they may contain schematic slots, which could be filled by various elements. For example, the idiom construction, “X take Y for granted”, has two schematic slots. They can be filled by various candidates: “You take him for granted”, “Many people take freedom for granted” (Hoffmann and Trousdale, 2013). Constructions do not leave their slots uncon- strained. They have a preference and restriction on how the slots are being fulfilled in text. The mea- sures of such constraints are extensively studied in the literature (Stefanowitsch, 2013). For example, one can compute the attraction and reliance of a word in a given construction (pattern) (Schmid, 2010). The attraction measures how a pattern prefers certain words to fill its slot; conversely, reliance indicates how a word tends to occur in particular constructions. A more detailed analysis may include the collostruction analysis, where the word attracted to a particular construction is re- ferred to collexeme. One can study the attraction and repulsion in the slots with collexeme analy- sis; and the interaction …

Content truncated to 500 words