Developing and Evaluating a Dataset for How-to Tip Machine Reading at Scale

Developing and Evaluating a Dataset for How-to Tip Machine Reading at Scale Fuzhu Zhu, Shuting Bai, Tingxuan Li, Takehito Utsuro Degree Programs in Systems and Information Engineering, Graduate School of Science and Technology, University of Tsukuba, 1-1-1, Tennodai, Tsukuba, Ibaraki, 305-8573, Japan {s2220804,s2020817,s2120816} @ s.tsukuba.ac.jp, utsuro @ iit.tsukuba.ac.jp Abstract In this paper, we focus on the task of machine reading at scale within how-to tip machine reading comprehension (MRC). We propose a method for developing a context dataset using how-to tip websites on the Internet as infor- mation sources. This shows that the proposed method can easily create a context dataset containing thousands of context sets. Fur- thermore, this paper uses a method for re- trieving the context from the developed con- text dataset, which contains the answer of the question. It applies to the MRC model. Specifically, we use three models based on TF-IDF and BERT (TF-IDF, BERT, and TF- IDF+BERT) as our retrieval models. Mean- while, the BERT model served as our MRC model. We apply the retrieval model and the MRC model to the context dataset after com- bining them. Evaluation results show that the TF-IDF+BERT model outperforms the other two models when tested against the context dataset. 1 Introduction In natural language processing, machine reading comprehension (MRC) tasks are formulated to ex- tract the answer to a question from a context within a few question sentences and contexts expressed in natural language. MRC tasks can be divided into two categories based on the two types of answers. Factoid MRC tasks aim at having the answer to fac- toids such as proper nouns and numbers, where the answer is usually unique, short and simple. Con- versely, nonfactoid MRC tasks aim to obtain an an- swer about nonfactoid such as explanation, reason and how-to tip, where there are usually multiple op- tions and the answer is frequently a full sentence, rahter than a word or phrase. The Stanford Ques- tion Answering Dataset (SQuAD) (Rajpurkar et al., 2016)is one of the most well-known QA datasets and benchmark tests among factoid MRC related to Wikipedia articles and news articles. Additionally, it is acknowledged that recent deep learning models (for example, BERT (Devlin et al., 2019)) trained with SQuAD achieved fairly high performance1. However, some research cases are known for non- factoid MRC. They include MS MARCO (Nguyen et al., 2016), which has been developed using Bing’s search logs and passages of retrieved web pages; DuReader (He et al., 2018), which has been de- veloped using Baidu Search; Baidu Zhidao, a Chi- nese community-based QA site; and the Narra- tiveQA (Koˇcisk´y et al., 2018) dataset (in English), which contains questions created by editors based on summaries of movie scripts and books. They also include Soleimani et al. (2021), Dulceanu et al. (2018), and Cohen et al. (2018). Among those working on nonfactoid MRC, the case of MRC of Japanese how-to tip QAs (Chen et al., 2020) selected the how-to tip websites that are posted on the Inter- net and chose the …

Content truncated to 500 words