Applying Masked Language Models to Search for Suitable Verbs Used in Academic Writing

Applying Masked Language Models to Search for Suitable Verbs Used in Academic Writing Chooi-Ling Goh The University of Kitakyushu Kitakyushu Fukuoka, Japan goh@kitakyu-u.ac.jp Abstract Non-native speakers of English tend to use in- appropriate expressions when relating to aca- demic writing. This may cause a non-fluent essay and probably unable to convey the pre- cise meaning of the texts. Therefore, it is im- portant to choose the best suitable vocabulary when writing an academic article, especially for international publication. This paper fo- cuses on word infilling on academic writing, where a word embedding model and some pre- trained masked language models are used to predict and fill in some blank verbs in the text like a cloze task. We evaluate the appropriate- ness of the predictions of the language mod- els by counting how many verbs can be pre- dicted exactly the same as the original texts and also the fluency of the outputs after re- placement. We conduct the test on two jour- nals in natural language processing field, one international and one local, in order to com- pare the fluency of texts written by native and non-native speakers of English. Our experi- ments show promising results and motivate us to design a learning and writing system that in- cludes both word embedding and masked lan- guage model features. 1 Introduction Recently, many advanced deep learning language models have been trained and proven to improve many natural language processing (NLP) tasks, such as text generation, summarization, machine trans- lation, question answering and etc. General pre- trained language models such as BERT (Devlin et al., 2018) and GPT-2 (Radford et al., 2019) are ac- tively used in these NLP tasks, and achieve excellent performance. In this research, we want to find out whether these language models can help non-native speakers to find the exact verbs used in academic writing. Non-native speakers of English tend to use inappropriate expressions when relating to academic writing. This will cause their essay to be unnatu- ral and non-fluent, and probably unable to convey the precise meaning of the texts. In the worst case, their papers might not be accepted for publication due to the low level of proficiency. Therefore, it is important to choose the best suitable vocabulary in academic writing, especially for international publi- cation. This paper focuses on word infilling on aca- demic writing, where pre-trained language models are used to fill in some blanks, in our research, the verbs in the text. BERT, a masked language model inspired by the cloze task, provides us a language model to in- vestigate how well it can fill in the blanks in the text in academic writing. Besides, a word embed- ding model such as Word2vec, can provide words that have close vectors for a specific word, which may have similar meaning or usage. We use these two types of language models to find suitable verbs used in texts for scientific field. BERT-based lan- guage models can predict verbs based on the con- text, whereas …

Content truncated to 500 words