Evaluating a How-to Tip Machine Comprehension Model with QA Examples collected from a Community QA Site

Evaluating a How-to Tip Machine Comprehension Model with QA Examples collected from a Community QA Site Tingxuan Li, Shuting Bai, Fuzhu Zhu, Takehito Utsuro Degree Programs in Systems and Information Engineering, Graduate School of Science and Technology, University of Tsukuba, 1-1-1, Tennodai, Tsukuba, Ibaraki, 305-8573, Japan Abstract In the field of factoid question answering (QA), it is known that the state-of-the-art tech- nology has achieved an accuracy comparable to human. However, in the area of non-factoid QA, there are only limited numbers of datasets for training QA models. So within the field of the non-factoid QA, Chen et al. (2020) de- veloped a dataset for training Japanese tip QA models. Although it can be shown that the trained Japanese tip QA model outperforms the factoid QA model, this paper further aims at answering tip questions more closely re- lated to daily lives. Specifically, we collect community QA examples from a community QA site and then apply the trained Japanese tip QA model to those community QA examples. Evaluation results show that the trained tip QA model outperforms the factoid QA model when testing against those community QA ex- amples. 1 Introduction Among factoid QA related to Wikipedia articles and news articles, the Stanford Question Answer- ing Dataset (SQuAD) (Pranav et al., 2016; Pranav et al., 2018) is one of the most well-known QA datasets and benchmark tests. SQuAD is a read- ing comprehension dataset. It consists of ques- tions posed by crowdworkers on a set of Wikipedia articles. The answer to every question is a text segment, or span, from the corresponding reading passage, or the question might be unanswerable. The state-of-the-art machine comprehension models trained with SQuAD outperform humans (Devlin et al., 2019; Zhang et al., 2019). However, there is a relatively limited number of published literature that handles the development of datasets for non-factoid QA and the application of state-of-the-art general- purpose machine comprehension models to those non-factoid datasets. Typical non-factoid QA tasks include opinion QA, definition QA, reason QA, and how-to tip QA. Among various kinds of non-factoid knowledge which are the key to developing techniques for non- factoid QA tasks, Chen et al. (2020) studied how to develop a dataset for training Japanese how-to tip (following Chen et al. (2020), we use the sim- plified term “tip”) QA models. As examples in Chen et al. (2020), tip QA datasets for ‘job hunting,” “marriage,” “apartment,” “hay fever,” “dentist,” and “food poisoning” are developed. “Job hunting” and “marriage” tip QAs are for both training and testing, while other tip QAs are only for testing. For “job hunting”, Figure 1 presents a typical example of a tuple of a context, a tip question, and an answer1. Chen et al. (2020) applied BERT (Devlin et al., 2019), one of the state-of-the-art machine compre- hension models, to a Japanese tip QA dataset. The trained tip QA model is also compared with a fac- toid QA model which is also trained with a Japanese factoid QA dataset. Evaluation results revealed that the …

Content truncated to 500 words