Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2129–2139, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2129 Multimodal Joint Attribute Prediction and Value Extraction for E-commerce Product Tiangang Zhu, Yue Wang, Haoran Li∗, Youzheng Wu, Xiaodong He and Bowen Zhou JD AI Research {zhutiangang3,wangyue274,lihaoran24,wuyouzheng1}@jd.com {xiaodong.he,bowen.zhou}@jd.com Abstract Product attribute values are essential in many e-commerce scenarios, such as customer ser- vice robots, product recommendations, and product retrieval. While in the real world, the attribute values of a product are usually incom- plete and vary over time, which greatly hinders the practical applications. In this paper, we propose a multimodal method to jointly pre- dict product attributes and extract values from textual product descriptions with the help of the product images. We argue that product at- tributes and values are highly correlated, e.g., it will be easier to extract the values on condi- tion that the product attributes are given. Thus, we jointly model the attribute prediction and value extraction tasks from multiple aspects to- wards the interactions between attributes and values. Moreover, product images have dis- tinct effects on our tasks for different prod- uct attributes and values. Thus, we selectively draw useful visual information from product images to enhance our model. We annotate a multimodal product attribute value dataset that contains 87,194 instances, and the experimen- tal results on this dataset demonstrate that ex- plicitly modeling the relationship between at- tributes and values facilitates our method to establish the correspondence between them, and selectively utilizing visual product infor- mation is necessary for the task. Our code and dataset are available at https://github. com/jd-aig/JAVE. 1 Introduction Product attribute values that provide details of the product are crucial parts of e-commerce, which help customers to make purchasing decisions and facilitate retailers on many applications, such as question answering system (Yih et al., 2015; Yu et al., 2017), product recommendations (Gong, ∗Corresponding author. This golden lapel shirt can be paired with black shoes …… “ ” Attribute Value Collar Type lapel Color golden Figure 1: An example of predicting attributes and ex- tracting values from the textual product description with the aid of the visual product information. 2009; Cao et al., 2018), and product retrieval (Liao et al., 2018; Magnani et al., 2019). While product attribute values are pervasively incomplete for a massive number of products on the e-commerce platform. According to our statistics on a main- stream e-commerce platform in China, there are over 40 attributes for the products in clothing cat- egory, but the average count of attributes present for each product is fewer than 8. The absence of the product attributes seriously affects customers’ shopping experience and reduces the potential of successful trading. In this paper, we propose a method to jointly predict product attributes and extract the corresponding values with multimodal product information, as shown in Figure 1. Though plenty of systems have been proposed to supplement product attribute values (Putthividhya and Hu, 2011; More, 2016; Shinzato and Sekine, 2013; Zheng et al., …

Content truncated to 500 words