Program Synthesis for Complex QA on Charts via Probabilistic Grammar Based Filtered Iterative Back-Translation

33262

Authors

nan

Abstract

AbstractAnswering complex reasoning questions from chart images is a challenging problem requiring a combination of natural language understanding, fine-grained perception, and analytical reasoning. Current chart-based Question Answering (QA) approaches largely address structural, visual or simple data retrieval-type questions with fixed-vocabulary answers and perform poorly on reasoning queries. We focus on answering realistic, complex, reasoning-based questions where the answer needs to be computed and not selected from a fixed set of choices. Our approach employs a neural semantic parser to transform Natural Language (NL) questions into SQL programs and execute them on a standardized schema populated from the extracted chart contents. In the absence of program annotations, i.e., in a weak supervision setting, we obtain initial SQL predictions from a pre-trained CodeT5 semantic parser and employ Filtered Iterative Back-Translation (FIBT) for iteratively augmenting our NL-SQL training set. The forward (neural semantic parser) and backward (language model) models are initially trained with an external NL-SQL dataset. We iteratively move towards the NL query distribution by generating NL questions from the synthesized SQL programs using a Probabilistic Context-Free Grammar (PCFG) where the production rule probabilities are induced to be inversely proportional to the probabilities in the training data. We filter out the generated NL queries with mismatched structures and compositions. Our FIBT approach achieves State-of-the-Art (SOTA) results on reasoning-based queries in the PlotQA dataset yielding a test accuracy of 60.44%, superseding the previous baselines by a large margin.

Keywords

nan

Full Content

Findings of the Association for Computational Linguistics: EACL 2023, pages 239–253 May 2-6, 2023 ©2023 Association for Computational Linguistics PePe: Personalized Post-editing Model utilizing User-generated Post-edits Jihyeon Lee*† gina.lee@kakaobrain.com Taehee Kim* taehee.kim@letsur.ai taeheekim@kaist.ac.kr Yunwon Tae*‡ yunwon.tae@vuno.co Cheonbok Park cbok.park@navercorp.com Jaegul Choo jchoo@kaist.ac.kr Abstract Incorporating personal preference is crucial in advanced machine translation tasks. Despite the recent advancement of machine translation, it remains a demanding task to properly reflect personal style. In this paper, we introduce a per- sonalized automatic post-editing framework to address this challenge, which effectively gen- erates sentences considering distinct personal behaviors. To build this framework, we first collect post-editing data that connotes the user preference from a live machine translation sys- tem. Specifically, real-world users enter source sentences for translation and edit the machine- translated outputs according to the user’s pre- ferred style. We then propose a model that com- bines a discriminator module and user-specific parameters on the APE framework. Experimen- tal results show that the proposed method out- performs other baseline models on four differ- ent metrics (i.e., BLEU, TER, YiSi-1, and hu- man evaluation). 1 Introduction Language usage is strongly influenced by the state of the individual, which can be considered by mul- tiple attributes such as age, gender, socioeconomic status, and occupation (Tannen et al., 1991; Pen- nebaker et al., 2003). Taking these aspects into account in the machine translation task, we need personalized translations to reflect individual char- acteristics that vary from person to person; thus, the translation system should consider not only fluency and content preservation, but also personal style. However, most existing neural machine transla- tion (NMT) models ignore personal style (Mirkin et al., 2015). Previous studies attempt to address this problem by personalizing the NMT models, but in these studies the definition of personal style * Indicates equal contribution. †Work done at Korea Advanced Institute of Science and Technology, correspondence to jihyeonlee@kaist.ac.kr. ‡Work done at Korea University, correspondence to tyj204@korea.ac.kr. Figure 1: Example of a personal post-editing triplet (i.e., source (src), machine translation (mt), and post-edit (pe)) given the source text in English and the translated text in Korean. A post-edited sentence does not only contain error correction of an initial machine transla- tion result but also reflects individual preference. For instance, a human post-editor modifies the word ”primar- ily” to ”primary,” but also change ”공헌” to its synonym ”기여” while keeping the rest as it is (e.g., ”research”). is often over-simplified (Rabinovich et al., 2017; Sennrich et al., 2016; Si et al., 2019). For example, Rabinovich et al. (2017) and Sennrich et al. (2016) define the personal style as politeness and gender respectively, which is not sufficient to tackle the multifarious character of an individual. Namely, previous works defined the personal style in a con- strained form. In contrast with previous studies, we propose a method based on an APE framework and newly utilize post-editing data to capture diverse personal traits in translation. Originally, the need for post- editing data is to improve the quality of machine- translated …

Content truncated to 500 words

Labeled Text

AbstractAnswering complex reasoning questions from chart images is a challenging problem requiring a combination of natural language understanding, fine-grained perception, and analytical reasoning. Current chart-based Question Answering (QA) approaches largely address structural, visual or simple data retrieval-type questions with fixed-vocabulary answers and perform poorly on reasoning queries. We focus on answering realistic, complex, reasoning-based questions where the answer needs to be computed and not selected from a fixed set of choices. Our approach employs a neural semantic parser to transform Natural Language (NL) questions into SQL programs and execute them on a standardized schema populated from the extracted chart contents. In the absence of program annotations, i.e., in a weak supervision setting, we obtain initial SQL predictions from a pre-trained CodeT5 semantic parser and employ Filtered Iterative Back-Translation (FIBT) for iteratively augmenting our NL-SQL training set. The forward (neural semantic parser) and backward (language model) models are initially trained with an external NL-SQL dataset. We iteratively move towards the NL query distribution by generating NL questions from the synthesized SQL programs using a Probabilistic Context-Free Grammar (PCFG) where the production rule probabilities are induced to be inversely proportional to the probabilities in the training data. We filter out the generated NL queries with mismatched structures and compositions. Our FIBT approach achieves State-of-the-Art (SOTA) results on reasoning-based queries in the PlotQA dataset yielding a test accuracy of 60.44%, superseding the previous baselines by a large margin.

Publication Details

Paper ID:
33262
Year:
2023
Publication:
TIPSTER
Publication Type:
conf
DOI:
nan