News Headline Grouping as a Challenging NLU Task

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3186–3198 June 6–11, 2021. ©2021 Association for Computational Linguistics 3186 News Headline Grouping as a Challenging NLU Task Philippe Laban UC Berkeley Lucas Bandarkar UC Berkeley Marti A. Hearst∗ UC Berkeley Abstract Recent progress in Natural Language Under- standing (NLU) has seen the latest models out- perform human performance on many stan- dard tasks. These impressive results have led the community to introspect on dataset lim- itations, and iterate on more nuanced chal- lenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corre- sponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a bi- nary judgement as to whether the pair belongs within the same group. On HLGD, human an- notators achieve high performance of around 0.9 F-1, while current state-of-the art Trans- former models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Gen- erator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high- performing models with consistency tests, and find that models are not consistent in their pre- dictions, revealing modeling limits of current architectures. 1 Introduction Headlines are a key component in everyday news consumption. As the first piece of text the user in- teracts with when learning about a story, the head- line can play many roles, including: summarize the main points of the story, promote a particular detail, and convince the reader to choose one source over another (Bonyadi and Samuel, 2013). News aggregators amass content from many dis- parate news sources and have become popular, in part because they offer news readers access to di- verse sources (Chowdhury and Landoni, 2006). Flaxman et al. (2016) find that news aggregators help news readers access content they are unfamil- iar with, and potentially on opposite sides of the ∗ Author emails: {phillab, lucasban- darkar,hearst}@berkeley.edu Date Headline Group 03/27 Russian-U.S.�crew�makes�belated arrival�at�space�station A 03/28 Russian�spacecraft�brings�3-man crew�to�ISS�after�2-day�delay A 03/29 Space�'makes�the�heart�grow�rounder' B 04/01 Astronauts'�hearts�become�spherical during�prolonged�trips�in�space,�study�finds B Figure 1: Snippet of timeline in the HeadLine Grouping dataset (HLGD). The headlines are part of the Space timeline, one of 10 timelines in HLGD. Head- lines labeled A are part of a group; those labeled B are part of another group. political spectrum. At the heart of a news aggrega- tor is the ability to group relevant content together, to support a reader in finding varying views and angles on the news. Natural Language Understanding (NLU) has seen rapid progress in recent years. The cre- ation of multi-task benchmarks such as the Gen- eral Language Understanding Evaluation collec- tion (GLUE), paired with fast-paced progress in Transformer-based architectures has led to mod- els outperforming human baseline performance on many tasks, such as paraphrase identification (Dolan et al., 2004), semantic similarity (Cer et al., 2017), and extractive question-answering (QA) (Rajpurkar et al., 2018). This success …

Content truncated to 500 words