A Distant Supervision Approach for Identifying Perspectives in Unstructured User-Generated Text

International Joint Conference on Natural Language Processing, pages 922–926, Nagoya, Japan, 14-18 October 2013. A Distant Supervision Approach for Identifying Perspectives in Unstructured User-Generated Text Attapol Thamrongrattanarit∗ Brandeis University Waltham, MA tet@brandeis.edu Colin Pollock Yelp Inc San Francisco, CA cpollock@yelp.com Benjamin Goldenberg Yelp Inc San Francisco, CA benjamin@yelp.com Jason Fennell Yelp Inc San Francisco, CA jfennell@yelp.com Abstract With the overabundance of online user- generated content, the ability to filter based on relevant perspectives is becom- ing increasingly important. Identifying the perspective of the authors with the review text would enhance the retrieval of perti- nent information. This problem can be tra- ditionally formulated as a text classifica- tion task and solved by annotating the data and building a supervised learning system. However, rare classes might render anno- tation even more difficult and expensive. Here, we used a distant supervision ap- proach to identify restaurant reviews that were written from the perspective of a veg- etarian, and we achieved a macro-average F1 score of 79.40% with minimal annota- tion effort. 1 Introduction The center of the information world has shifted from select few authorities to the global wisdom of the crowd and user-generated content. While useful and large, the volume of the information re- quires efficient organization, information retrieval, and data mining techniques to select the most rele- vant contents. For example, restaurant goers look- ing for vegetarian-friendly restaurants might want to read restaurant reviews that are written by a veg- etarian. Authors’ perspectives potentially provide a meaningful axis along which the documents can be organized. Past studies have formulated this problem as a document-level supervised text classification ∗The author conducted the work during his internship at Yelp. problem (Manning et al., 2008), but the super- vised learning paradigm might not be suitable in certain scenarios. Supervised learning algorithms can achieve superior performance compared to unsupervised learning algorithms at the expense of costly annotation efforts in creating labeled datasets for the algorithms to learn from. The per- spectives that we would like to identify, however, might be very specific and somewhat rare in the document collection. To continue with the restau- rant review example, only an estimated of 3.2% of the U.S. population identify themselves as vege- tarian (Haddad and Tanzman, 2003), so the restau- rant reviews written from the perspective of a veg- etarian might be very rare in the corpus. This rare class problem necessitates a larger number of annotated documents to collect suffi- cient positive examples. For instance, approxi- mately 10,000 data instances must be labeled in order to obtain a mere 320 data points for vegetar- ian reviews. Additionally, the resulting classifier trained on a specific annotated corpus will tend to be biased toward that text domain, and the perfor- mance might degenerate when the classifier is ap- plied to text in another domain. Although the rare class problem is well studied in supervised settings (He and Ghodsi, 2010; Joshi et al., 2001), to our knowledge we have not encountered a distantly su- pervised algorithm applied to …

Content truncated to 500 words