Assessing the Helpfulness of Learning Materials with Inference-Based Learner-Like Agent

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3807–3817, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3807 Assessing the Helpfulness of Learning Materials with Inference-Based Learner-Like Agent Yun-Hsuan Jen1, Chieh-Yang Huang2, Mei-Hua Chen3, Ting-Hao (Kenneth) Huang2, and Lun-Wei Ku1 1Academia Sinica, Taipei, Taiwan. yhjen2@gmail.com, lwku@iis.sinica.edu.tw 2Pennsylvania State University, University Park, PA, USA. {chiehyang,txh710}@psu.edu 3Tunghai University, Taichung, Taiwan. mhchen@thu.edu.tw Abstract Many English-as-a-second language learners have trouble using near-synonym words (e.g., small vs. little; briefly vs. shortly) cor- rectly, and often look for example sentences to learn how two nearly synonymous terms dif- fer. Prior work uses hand-crafted scores to rec- ommend sentences but have difficulty in adopt- ing such scores to all the near-synonyms as near-synonyms differ in various ways. We no- tice that the helpfulness of the learning ma- terial would reflect on the learners’ perfor- mance. Thus, we propose the inference-based learner-like agent to mimic learner behavior and identify good learning materials by exam- ining the agent’s performance. To enable the agent to behave like a learner, we leverages entailment modeling’s capability of inferring answers from the provided materials. Exper- imental results show that the proposed agent is equipped with good learner-like behavior to achieve the best performance in both fill-in- the-blank (FITB) and good example sentence selection tasks. We further conduct a class- room user study with college ESL learners. The results of the user study show that the pro- posed agent can find out example sentences that help students learn more easily and effi- ciently. Compared to other models, the pro- posed agent improves the score of more than 17% of students after learning. 1 Introduction Many English-as-a-second-language (ESL) learn- ers have trouble using near-synonyms cor- rectly (Liu and Zhong, 2014; Liu, 2013). “Near- synonym” refers to a word whose meaning is sim- ilar but not identical to that of another word, for instance, establish and construct. An experience common to many ESL learners is looking for exam- ple sentences to learn how two nearly synonymous words differ (Liu, 2013; Liu and Jiang, 2009). To facilitate the learner’s learning process, our focus Figure 1: The Learner-Like Agent mimics learners’ be- havior of performing well when learning from good ma- terial and vice versa. We utilize such a behavior to find out helpful learning materials. is on finding example sentences to clarify English near-synonyms. In previous work, researchers develop linguis- tic search engines, such as Linggle (Boisson et al., 2013) and Netspeak1, to allow users to query En- glish words in terms of n-gram frequency. However, these tools can only help people investigate the dif- ference, where learners are required to make as- sumptions toward the subtlety and verify them with the tools, but can not tell the difference proactively. Other work attempts to automatically retrieve ex- ample sentences for dictionary entries (Kilgarriff et al., 2008); however, finding clarifying examples for near-synonyms is not the goal of such work. In a rare exception, Huang et al. (2017) retrieve useful examples for near-synonyms …

Content truncated to 500 words