I Wish I Would Have Loved This One, But I Didn’t – A Multilingual Dataset for Counterfactual Detection in Product Review

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7092–7108 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 7092 I Wish I Would Have Loved This One, But I Didn’t – A Multilingual Dataset for Counterfactual Detection in Product Reviews James O’Neill‡∗ James.O-Neill@liverpool.ac.uk Polina Rozenshtein†,∗ prrozens@amazon.co.jp Ryuichi Kiryo† kiryor@amazon.co.jp Motoko Kubota† kubmotok@amazon.co.jp Amazon†, University of Liverpool‡ Danushka Bollegala†,‡ danubol@amazon.com Abstract Counterfactual statements describe events that did not or cannot take place. We consider the problem of counterfactual detection (CFD) in product reviews. For this purpose, we an- notate a multilingual CFD dataset from Ama- zon product reviews covering counterfactual statements written in English, German, and Japanese languages. The dataset is unique as it contains counterfactuals in multiple lan- guages, covers a new application area of e- commerce reviews, and provides high quality professional annotations. We train CFD mod- els using different text representation meth- ods and classifiers. We find that these mod- els are robust against the selectional biases in- troduced due to cue phrase-based sentence se- lection. Moreover, our CFD dataset is com- patible with prior datasets and can be merged to learn accurate CFD models. Applying ma- chine translation on English counterfactual ex- amples to create multilingual data performs poorly, demonstrating the language-specificity of this problem, which has been ignored so far. 1 Introduction Counterfactual statements are an essential tool of human thinking and are often found in natural lan- guages. Counterfactual statements may be identi- fied as statements of the form – If p was true, then q would be true (i.e. assertions whose antecedent (p) and consequent (q) are known or assumed to be false) (Milmed, 1957). In other words, a coun- terfactual statement describes an event that may not, did not, or cannot take place, and the subse- quent consequence(s) or alternative(s) did not take place. For example, consider the counterfactual statement – I would have been content with pur- chasing this iPhone, if it came with a warranty!. Counterfactual statements can be broken into two parts: a statement about the event (if it came with a warranty), also referred to as the antecedent, and ∗The two first authors contributed equally the consequence of the event (I would have been content with purchasing this iPhone), referred to as the consequent. Counterfactual statements are ubiquitous in natural language and have been well- studied in fields such as philosophy (Lewis, 2013), psychology (Markman et al., 2007; Roese, 1997), linguistics (Ippolito, 2013), logic (Milmed, 1957; Quine, 1982), and causal inference (H¨ofler, 2005). Accurate detection of counterfactual statements is beneficial to numerous applications in natural language processing (NLP) such as in medicine (e.g., clinical letters), law (e.g., court proceedings), sentiment analysis, and information retrieval. For example, in information retrieval, counterfactual detection (CFD) can potentially help to remove ir- relevant results to a given query. Revisiting our previous example, we should not return the iPhone in question for a user who is searching for iPhone with warranty because that iPhone does not come with a warranty. …

Content truncated to 500 words