Does Joint Training Really Help Cascaded Speech Translation?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 18 - 29 December 7-11, 2022 ©2022 Association for Computational Linguistics CDCONV: A Benchmark for Contradiction Detection in Chinese Conversations Chujie Zheng1∗ Jinfeng Zhou1,2∗ Yinhe Zheng3 Libiao Peng3 Zhen Guo4 Wenquan Wu4 Zheng-Yu Niu4 Hua Wu4 Minlie Huang1,3† 1The CoAI Group, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, 1Beijing National Research Center for Information Science and Technology, DCST, Tsinghua University, Beijing 100084, China 2College of Intelligence and Computing, Tianjin University, Tianjin, China 3Lingxin AI, Beijing 100084, China 4Baidu Inc., China chujiezhengchn@gmail.com jfzhou.mail@gmail.com aihuang@tsinghua.edu.cn {guozhenguozhen, wuwenquan01, niuzhengyu, wu_hua}@baidu.com Abstract Dialogue contradiction is a critical issue in open-domain dialogue systems. The con- textualization nature of conversations makes dialogue contradiction detection rather chal- lenging. In this work, we propose a bench- mark for Contradiction Detection in Chinese Conversations, namely CDCONV. It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intra- sentence Contradiction, Role Confusion, and History Contradiction. To efficiently construct the CDCONV conversations, we devise a series of methods for automatic conversation gener- ation, which simulate common user behaviors that trigger chatbots to make contradictions. We conduct careful manual quality screening of the constructed conversations and show that state-of-the-art Chinese chatbots can be eas- ily goaded into making contradictions. Exper- iments on CDCONV show that properly mod- eling contextual information is critical for di- alogue contradiction detection, but there are still unresolved challenges that require future research.1 1 Introduction Large-scale pre-training for dialogue generation (Zhang et al., 2020; Freitas et al., 2020) has ad- vanced the development of engaging and human- like dialogue systems. Unfortunately, state-of- the-art open-domain chatbots, such as BlenderBot (Roller et al., 2021), EVA (Zhou et al., 2021; Gu et al., 2022) and PLATO (Bao et al., 2021b), still of- ten behave inconsistently with their role or identity and produce utterances that are self-contradictory ∗Equal contribution. †Corresponding author. 1Our data and codes are available at https: //www.github.com/thu-coai/CDConv and https://github.com/PaddlePaddle/Knover/ tree/dygraph/projects/cdconv Non-contradiction 𝑢1: 你喜欢吃面条吗？(Do you like noodles?) 𝑏1: 我喜欢吃米饭! (I love to eat rice!) 𝑢2: 你不喜欢面条吗? (Don’t you like noodles?) 𝑏2: 不喜欢! (I dislike!) Contradiction 𝑢1: 那你喜欢什么小动物呢? (What animals do you like?) 𝑏1: 狗,猫 (Dogs, cats) 𝑢2: 你不喜欢什么小动物? (What animals do you dislike?) 𝑏2: 不喜欢猫,其他的都喜欢 (I dislike cats. I like all the other animals) {"model": "plato", "method": "反义-否定词", "u1": "那你喜欢看什么类型的电影？", " {"model": "plato", "method": "反义-否定词", "u1": "那你喜欢看什么类型的电影？", " {"model": "plato", "method": "反义-否定词", "u1": "那你喜欢看什么类型的电影？", " {"model": "plato", "method": "反义-否定词", "u1": "你难道不喜欢她吗？", "b1": "不喜 {"model": "plato", "method": "反义-否定词", "u1": "你难道不喜欢她吗？", "b1": "不喜 {"model": "plato", "method": "反义-否定词", "u1": "你难道不喜欢她吗？", "b1": "不喜 {"model": "plato", "method": "反义-否定词", "u1": "你难道不喜欢她吗？", "b1": "不喜 {"model": "plato", "method": "反义-否定词", "u1": "你喜欢干什么平时？", "b1": "玩游 {"model": "eva", "method": "反义-否定词", "u1": "那你下班之后最喜欢干的事是什么 Figure 1: Dialogue contradiction detection requires the full contextual information (including u1 and u2) rather than only the bot’s utterances (i.e., b1 and b2). or contradict the dialogue history (Shuster et al., 2022; Gu et al., 2022; Xu et al., 2022a). Such in- consistency or contradiction phenomena violate Grice’s cooperative …

Content truncated to 500 words