On the Branching Bias of Syntax Extracted from Pre-trained Language Models

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4473–4478 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4473 On the Branching Bias of Syntax Extracted from Pre-trained Language Models Huayang Li Lemao Liu Guoping Huang Shuming Shi Tencent AI Lab {alanili,redmondliu,donkeyhuang,shumingshi}@tencent.com Abstract Many efforts have been devoted to extracting constituency trees from pre-trained language models, often proceeding in two stages: fea- ture definition and parsing. However, this kind of methods may suffer from the branching bias issue, which will inflate the performances on languages with the same branch it biases to. In this work, we propose quantitatively measur- ing the branching bias by comparing the per- formance gap on a language and its reversed language, which is agnostic to both language models and extracting methods. Furthermore, we analyze the impacts of three factors on the branching bias, namely parsing algorithms, feature definitions, and language models. Ex- periments show that several existing works ex- hibit branching biases, and some implementa- tions of these three factors can introduce the branching bias. 1 Introduction Neural language models such as LSTM (Merity et al., 2018; Peters et al., 2018), GPT2 (Radford et al., 2019), and BERT (Devlin et al., 2019; Liu et al., 2019) have achieved state-of-the-art perfor- mance in various downstream NLP tasks. Many recent works try to interpret their success by reveal- ing the linguistic properties captured by these lan- guage models (Hewitt and Manning, 2019; Clark et al., 2019; Jawahar et al., 2019; Tenney et al., 2019). One interesting line of these works tries to extract discrete constituency trees from pre-trained language models (Mareˇcek and Rosa, 2018, 2019; Kim et al., 2020; Wu et al., 2020). The core of these works is to extract syntax in two stages. Firstly, it defines the feature scores based on a language model, namely, the feature definition stage. Sec- ondly, it leverages the feature scores to build a constituency tree, namely, the parsing stage. However, the degree to which the extracted con- stituency trees match gold constituency annotations The test may come today The test may come today Figure 1: Constituency trees of a right-branching lan- guage and its reversed (left-branching) language. The tree at the bottom is obtained by reversing the tree at the top. may imprecisely reflect the model’s competence of capturing syntax, since their final performance may benefit from the branching bias. For exam- ple, as pointed out by Dyer et al. (2019), the syn- tax extracted from the ordered neuron based lan- guage model (Shen et al., 2019) is biased to right- branching languages 1 (e.g., English). Nevertheless, the approach to measuring the bias in Dyer et al. (2019) is highly dependent on the architecture of ordered neuron and its parsing algorithm. There- fore, it is far from trivial to be applied to general pre-trained language models. This paper proposes a new approach to reveal the branching bias of syntax from pre-trained language models, which is agnostic to model architectures and parsing algorithms. The key idea of …

Content truncated to 500 words