A Dataset and Baselines for Multilingual Reply Suggestion

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 1207–1220 August 1–6, 2021. ©2021 Association for Computational Linguistics 1207 A Dataset and Baselines for Multilingual Reply Suggestion Mozhi Zhang∗ University of Maryland mozhi@cs.umd.edu Wei Wang† Qualtrics wwang@qualtrics.com Budhaditya Deb Microsoft AI budeb@microsoft.com Guoqing Zheng Microsoft Research zheng@microsoft.com Milad Shokouhi Microsoft AI milads@microsoft.com Ahmed Hassan Awadallah Microsoft Research hassanam@microsoft.com Abstract Reply suggestion models help users process emails and chats faster. Previous work only studies English reply suggestion. Instead, we present MRS, a multilingual reply suggestion dataset with ten languages. MRS can be used to compare two families of models: 1) re- trieval models that select the reply from a fixed set and 2) generation models that produce the reply from scratch. Therefore, MRS com- plements existing cross-lingual generalization benchmarks that focus on classification and sequence labeling tasks. We build a gener- ation model and a retrieval model as base- lines for MRS. The two models have differ- ent strengths in the monolingual setting, and they require different strategies to generalize across languages. MRS is publicly available at https://github.com/zhangmozhi/mrs. 1 Multilingual Reply Suggestion Automated reply suggestion is a useful feature for email and chat applications. Given an input mes- sage, the system suggests several replies, and users may click on them to save typing time (Figure 1). This feature is available in many applications in- cluding Gmail, Outlook, LinkedIn, Facebook Mes- senger, Microsoft Teams, and Uber. Reply suggestion is related to but different from open-domain dialog systems or chatbots (Adiwar- dana et al., 2020; Huang et al., 2020). While both are conversational AI tasks (Gao et al., 2019), the goals are different: reply suggestion systems help the user quickly reply to a message, while chatbots aim to continue the conversation and focus more on multi-turn dialogues. Ideally, we want our model to generate replies in any language. However, reply suggestion models require large training sets, so previous work mostly ∗Work mostly done as an intern at Microsoft Research. †Work done at Microsoft Research. Figure 1: An example of reply suggestion system. User can click on the suggestions for a quick reply. focuses on English (Kannan et al., 2016; Henderson et al., 2017; Deb et al., 2019). To investigate reply suggestion for other languages with possibly lim- ited data, we build a multilingual dataset, dubbed MRS (Multilingual Reply Suggestion). From pub- licly available Reddit threads, we extract message- reply pairs, response sets, and machine-translated examples in ten languages (Table 1). One interesting aspect of the reply suggestion problem is that there are two modeling approaches. Some models follow the retrieval framework and select the reply from a predetermined response set (Henderson et al., 2017). Others follow the generation framework and generate the reply from scratch (Kannan et al., 2016). The two approaches have different advantages. Generation models are more powerful because they are not constrained by the response set. In comparison, retrieval models are easier to train and runs …

Content truncated to 500 words