Monitio - Large Scale MT for Multilingual Media Monitoring

Monitio – Large Scale MT for Multilingual Media Monitoring Carlos Amaral Priberam Lisbon, Portugal carlos@priberam.pt Sebastião Miranda Priberam Lisbon, Portugal ssm@priberam.pt Abstract Monitio is a real-time crosslingual global media monitoring platform which deliv- ers actionable insights beyond human scale and capabilities. Our system con- tinuously ingests a massive number of multilingual data sources that are auto- matically translated, filtered and catego- rized to generate intelligence reports spe- cially geared towards media monitoring professionals’ needs. 1 Origin The starting point of Monitio was a multilingual media monitoring prototype developed between 2016–2019 in tight collaboration with the British Broadcast Corporation (BBC) and Deutsche Welle (DW).2 Both broadcasters monitor a grow- ing number of video streams in different lan- guages, by assigning teams of human analysts that are grouped by languages. This approach is not scalable hence the need to reinvent media monitoring, tackling it globally and in a scalable fashion to break the current internationalization and scalability barriers. The emergence of mature natural language processing (NLP) and artificial intelligence tech- nologies gives European companies an oppor- tunity to push Europe to the leadership of the media monitoring market, where multilinguality is a major issue and, simultaneously, a major opportunity. © 2022 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CCBY-ND. 2 http://summa-project.eu By integrating machine translation (MT) in the ingestion and enrichment pipeline, Monitio ena- bles the monitoring of sources in languages the human analyst is not fluent in, providing a truly global view of the events not culturally, geo- graphically or politically biased for lack of ac- cess to a broader set of sources. 2 Challenges A major challenge for Monitio is the huge volume of data ingested daily into the platform. One of Monitio’s goals is to process 10 million new multilingual items per day. This has to be accomplished with a minimum delay to enable near real-time monitoring. The automatic translation of this amount of documents is just one of the challenges for the platform’s enrichment pipeline. All content entering the platform is subject to a series of NLP steps, namely its classification according to a standard topics taxonomy, the recognition of named entities like people, organizations, brands and places and linking them to external knowledge bases like Wikipedia or Wikidata, the production of a summary, and the clustering of all articles related to the same event in storylines. Adding to the complexity, the large-scale dis- semination of content on social media platforms, while ensuring a broad coverage of multiple con- nected viewpoints on the same subject of inter- est, stresses the problem of information verifica- tion which is a daunting task without the support of automation. Failing to address this problem leads to biased views on the subject and insuffi- cient (or even wrong) insights. Figure 1: Multilingual Storyline 3 Solution The scalability goals but also the cost implica- tions led to the integration of free/open source MT models like EasyNMT from Hugging Face, deployed in a dedicated GPU infrastructure. …

Content truncated to 500 words