Modeling Exemplification in Long-form Question Answering via Retrieval

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2079 - 2092 July 10-15, 2022 ©2022 Association for Computational Linguistics Modeling Exemplification in Long-form Question Answering via Retrieval Shufan Wang1, Fangyuan Xu2, Laure Thompson1, Eunsol Choi2, Mohit Iyyer1 1 College of Information and Computer Sciences, University of Massachusetts Amherst 2 Department of Computer Science, The University of Texas at Austin {shufanwang, laurejt, miyyer}@cs.umass.edu {fangyuan, eunsol}@utexas.edu Abstract Exemplification is a process by which writers explain or clarify a concept by providing an example. While common in all forms of writ- ing, exemplification is particularly useful in the task of long-form question answering (LFQA), where a complicated answer can be made more understandable through simple examples. In this paper, we provide the first computational study of exemplification in QA, performing a fine-grained annotation of different types of ex- amples (e.g., hypotheticals, anecdotes) in three corpora. We show that not only do state-of-the- art LFQA models struggle to generate relevant examples, but also that standard evaluation met- rics such as ROUGE are insufficient to judge exemplification quality. We propose to treat ex- emplification as a retrieval problem in which a partially-written answer is used to query a large set of human-written examples extracted from a corpus. Our approach allows a reliable ranking- type automatic metrics that correlates well with human evaluation. A human evaluation shows that our model’s retrieved examples are more relevant than examples generated from a state- of-the-art LFQA model. 1 Introduction When an author introduces a complicated concept, they commonly follow it up with a concrete exam- ple to help clarify their intended meaning. This pro- cess, known as exemplification, occurs in diverse forms, including hypothetical examples, personal anecdotes, and analogies (Clouse, 2013). Exempli- fication is particularly common within the NLP task of long-form question answering (LFQA), where an author wants to communicate a concept un- known to the question asker. Consider the follow- ing QA pair from the r/explainlikeimfive subreddit (Fan et al., 2019): Q: How does the ground not cave in while being under the heavy weight of cities? A: It’s all about what they’re building on, and occasionally they get it wrong... For example, San Francisco’s Millennium Tower, built on mud and sand, has already sunk 18 inches into the ground. Here, the answerer uses a specific example to emphasize the importance of building on a solid foundation. In general, explaining via exemplifica- tion is a fundamental technique in these kinds of pedagogical scenarios (Hyland, 2007), and it war- rants separate study due to its importance in LFQA and the challenges in evaluating it. However, ex- isting work on building LFQA models (Fan et al., 2019) does not give special treatment to exemplifi- cation or any other discourse phenomena, choosing instead to evaluate model outputs against reference answers using metrics such as ROUGE (Lin, 2004) that are not meaningful for this task (Krishna et al., 2021). In the above QA pair, any other structurally- unstable building …

Content truncated to 500 words