Chat-log Disentanglement via Same-Thread Classification and Direct-Reply Prediction

Chat-log Disentanglement via Same-Thread Classification and Direct-Reply Prediction Chia-Hui Chang, Zhi-Xian Liu Yu-Ching Liao, Yu-Hao Wu National Central University, Taiwan chia@csie.ncu.edu.tw Thamolwan Poopradubsil Department of Computer Science Kasetsart University, Thailand tmw.poopradubsil@gmail.com Abstract The early purpose of chatlog (conversation) dis- entanglement is to separate intermingled mes- sages into detached conversations for easier in- formation following and relevant information retrieving from simultaneous messages. Thus, the problem has been modeled as predicting whether two messages come from the same- thread. While the previous study by (Jiang et al., 2018) seems to perform well on same- thread prediction, we find that it is because the data are randomly split into training and test sets, resulting overlapping of topics in training and testing sets. When data is split by time order, the performance of existing models drop significantly. In this study, we consider the problem of direct reply predication task and study different message pair classification mod- els for the task. We argue that independent mes- sage encoders could better represent messages to capture their interaction than shared message encoders especially for direct-reply prediction task. We also find that BERT model performs well with small datasets, while other models may outperform BERT with large datasets. 1 Introduction With the continuous development of the Internet and social media, online group discussions and conver- sations have become increasingly popular and play an important role in society and the economy. Many commercial sites take advantage of this to advertise their products or help solve users’ problems. For example, developing conversational agents (chatbots) on their website to help staffs answer questions that may have been asked before. The goal of chatlog disentanglement is to cluster messages that belong to the same topic for tracing. Selecting a reply for a given input or finding question- answer pairs are special cases of conversation disen- tanglement. For example, Figure 1 shows a segment of conversations consisting of four ongoing threads in the IRC (Internet Relay Chat) conversation. As we can see, interleaved conversations can occur in both two-person or multi-person chats. Thus, the goal of conversation disentanglement is to match the message pairs for question-answer pair generation. There are many studies on conversation disentan- glement in the past. One solution for conversation disentanglement is to model the topic of messages by estimating the similarity between messages and decide whether each incoming message starts a new topic or belong to an existing thread. Jiang et al. (2018) proposed Siamese Hierarchi- cal Convolutional Neural Network (SHCNN) which integrates two hierarchical CNNs to capturing low- order and high-order semantics of the messages for a better message representation. By concatenating ab- solute difference of the two representation with other temporal and user information, SHCNN predicts the probability of two messages belonging to the same- thread with high accuracy on the data of IRC and Reddit. However, the model performs poorly for fu- ture unseen messages when the training and testing data are split by time order. We argue that the proposed data preparation method by (Jiang …

Content truncated to 500 words