Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 40–47 July 13-14, 2023 ©2023 Association for Computational Linguistics Analyzing Syntactic Generalization Capacity of Pre-trained Language Models on Japanese Honorific Conversion Ryo Sekizawa and Hitomi Yanaka The University of Tokyo {ryosekizawa,hyanaka}@is.s.u-tokyo.ac.jp Abstract Using Japanese honorifics is challenging be- cause it requires not only knowledge of the grammatical rules but also contextual informa- tion, such as social relationships. It remains un- clear whether pre-trained large language mod- els (LLMs) can flexibly handle Japanese hon- orifics like humans. To analyze this, we intro- duce an honorific conversion task that considers social relationships among people mentioned in a conversation. We construct a Japanese honorifics dataset from problem templates of various sentence structures to investigate the syntactic generalization capacity of GPT-3, one of the leading LLMs, on this task under two settings: fine-tuning and prompt learning. Our results showed that the fine-tuned GPT-3 per- formed better in a context-aware honorific con- version task than the prompt-based one. The fine-tuned model demonstrated overall syn- tactic generalizability towards compound hon- orific sentences, except when tested with the data involving direct speech. 1 Introduction The correct use of Japanese honorifics is difficult because it requires both the knowledge of gram- matical rules (i.e., verb conjugation) and contex- tual information (i.e., social relationships among the speaker, the hearer, and the people mentioned in a conversation) (Harada, 1976). We expect this syntactic and pragmatic ability for pre-trained large language models (LLMs), as they have shown high performance on natural language tasks (Brown et al., 2020, Ouyang et al., 2022, inter alia). How- ever, it remains unclear whether LLMs can handle Japanese honorifics in a similar manner to humans, based on sentence structures and social context. Several studies proposed datasets of Japanese honorifics for classification (Liu and Kobayashi, 2022; Someya and Oseki, 2022) and genera- tion (Matsumoto et al., 2022). Liu and Kobayashi (2022) introduced a task in which a model takes an honorific sentence as input and classifies its hon- orific level or the types of honorifics used in the sentence. Someya and Oseki (2022) provided a Japanese acceptability classification dataset called JCoLA. In JCoLA, subject honorifics are catego- rized as sub-categories of subject-verb agreement tasks. However, these datasets aim to evaluate the syntactic performance of language models, and they do not analyze their pragmatic ability to under- stand honorifics by considering social relationships behind sentences. Matsumoto et al. (2022) intro- duced an evaluation dataset for an honorific con- version task in which the input was a non-honorific sentence, and the output was an honorific sentence. Matsumoto et al. (2022) mentioned the necessity of considering the information on social relationships among people in honorific conversion but did not clarify how such information should be processed in the task. In summary, the existing benchmark datasets of Japanese honorifics focus on either syn- tactic or pragmatic knowledge required for hon- orific understanding, not both (Appendix A). Ad- ditionally, none of these existing studies discusses the generalization …

Content truncated to 500 words