On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART

Proceedings of the 29th International Conference on Computational Linguistics, pages 6516–6529 October 12–17, 2022. 6516 On the Role of Pre-trained Language Models in Word Ordering: A Case Study with BART Zebin Ou1, Meishan Zhang3, Yue Zhang1,2∗ 1School of Engineering, Westlake University 2Institute of Advanced Technology, Westlake Institute for Advanced Study 3Institute of Computing and Intelligence, Harbin Institute of Technology (Shenzhen) {ouzebin,zhangyue}@westlake.edu.cn, mason.zms@gmail.com Abstract Word ordering is a constrained language gen- eration task taking unordered words as input. Existing work uses linear models and neural networks for the task, yet pre-trained language models have not been studied in word ordering, let alone why they help. We use BART as an instance and show its effectiveness in the task. To explain why BART helps word ordering, we extend analysis with probing and empiri- cally identify that syntactic dependency knowl- edge in BART is a reliable explanation. We also report performance gains with BART in the related partial tree linearization task, which readily extends our analysis.1 1 Introduction The task of word ordering (Wan et al., 2009; Zhang and Clark, 2015; Tao et al., 2021), also known as linearization (Liu et al., 2015), aims to assign a valid permutation to a bag of words for a coherent sentence. While early work uses word ordering to improve the grammaticality of machine-generated sentences (Wan et al., 2009), the task subsequently manifests itself in applications such as discourse generation (Althaus et al., 2004), machine trans- lation (Tromble and Eisner, 2009; He and Liang, 2011), and image captioning (Fang et al., 2015). It plays a central role in linguistic realization (Gatt and Krahmer, 2018) of pipeline text generation systems. Advances in word ordering are also rele- vant to retrieval augmented generation (Guu et al., 2020), with outputs additionally conditioned on re- trieved entries, which can constitute a bag of words. Word ordering can be viewed as constrained lan- guage generation with all inflected output words provided, which makes it more amenable for error analysis (§3.4). The task can be extended to tree linearization (He et al., 2009) or partial tree lin- earization (Zhang, 2013) with syntactic features as ∗Corresponding Author 1We release our source code at https://github. com/simtony/BART-word-orderer <latexit sha1_base64="OEWOIhvxhVaBPxmAsNxEgB41ME=">AB9HicbVDL SgNBEJyNrxhfUY9eBoPgKexKUI8BL+IpgnlAsoTZScZMju7zvQGw5Lv8OJBEa9+jDf/xkmyB0saCiqunuCmIpDLrut5NbW9/Y3MpvF3Z29/YPiodHDRMlmkOdRzL SrYAZkEJBHQVKaMUaWBhIaAajm5nfHIM2IlIPOInBD9lAib7gDK3kdxCeEDG9i4Zq2i2W3LI7B10lXkZKJEOtW/zq9CKehKCQS2ZM23Nj9FOmUXAJ0InMRAzPmIDaF uqWAjGT+dHT+mZVXq0H2lbCulc/T2RstCYSRjYzpDh0Cx7M/E/r51g/9pPhYoTBMUXi/qJpBjRWQK0JzRwlBNLGNfC3kr5kGnG0eZUsCF4y+vksZF2bsV+4rpWoli yNPTsgpOSceuSJVcktqpE4eSTP5JW8OWPnxXl3PhatOSebOSZ/4Hz+AHovkoc=</latexit>John <latexit sha1_base64="LvQXjFY3AhQ/vQjLbOVtN3sZhQs=">AB83icbVBN S8NAEN34WetX1aOXxSJ4KokU9Vjw4rGC/YAmlM120i7dbMLuRCyhf8OLB0W8+me8+W/ctjlo64OBx3szMwLUykMu63s7a+sbm1Xdop7+7tHxWjo7bJsk0hxZPZK 7ITMghYIWCpTQTWwOJTQCce3M7/zCNqIRD3gJIUgZkMlIsEZWsn3EZ4QMWcI036l6tbcOegq8QpSJQWa/cqXP0h4FoNCLpkxPc9NMciZRsElTMt+ZiBlfMyG0LNUsR hMkM9vntJzqwxolGhbCulc/T2Rs9iYSRzazpjhyCx7M/E/r5dhdBPkQqUZguKLRVEmKSZ0FgAdCA0c5cQSxrWwt1I+YpxtDGVbQje8surpH1Z865q9ft6tVEv4iRU 3JGLohHrkmD3JEmaRFOUvJMXsmbkzkvzrvzsWhdc4qZE/IHzucPz4uSKA=</latexit>ate <latexit sha1_base64="OVOCeUzGp6rXMjXjesRK2BX/f4=">AB8nicbVBN S8NAEJ34WetX1aOXYBE8lUSKeix48VjBfkAayma7aZdudsPuRCyhP8OLB0W8+mu8+W/ctjlo64OBx3szMyLUsENet63s7a+sbm1Xdop7+7tHxWjo7bRmWashZVQul uRAwTXLIWchSsm2pGkiwTjS+nfmdR6YNV/IBJykLEzKUPOaUoJWCHrInRMyJnPYrVa/mzeGuEr8gVSjQ7Fe+egNFs4RJpIYE/heimFONHIq2LTcywxLCR2TIQslS RhJsznJ0/dc6sM3FhpWxLdufp7IieJMZMksp0JwZFZ9mbif16QYXwT5lymGTJF4viTLio3Nn/7oBrRlFMLCFUc3urS0dE4o2pbINwV9+eZW0L2v+Va1+X6826kUcJ TiFM7gAH6hAXfQhBZQUPAMr/DmoPivDsfi9Y1p5g5gT9wPn8A6ORsw=</latexit>an <latexit sha1_base64="YviMqBfMadj/fGhCjY5wi/CcmIc=">AB9XicbVBN SwMxEM36WetX1aOXYBE8lV0p6rHgxWMF+wHtWrLpbBuazS7JrFqW/g8vHhTx6n/x5r8xbfegrQ8GHu/NJDMvSKQw6Lrfzsrq2vrGZmGruL2zu7dfOjhsmjVHBo8lrF uB8yAFAoaKFBCO9HAokBCKxhdT/3WA2gjYnWH4wT8iA2UCAVnaKX7LsITImYsSRMeqWyW3FnoMvEy0mZ5Kj3Sl/dfszTCBRyYzpeG6CfsY0Cm7fK3ZTAwnjIzaAjq WKRWD8bLb1hJ5apU/DWNtSGfq74mMRcaMo8B2RgyHZtGbiv95nRTDKz8TKkRFJ9/FKaSYkynEdC+0MBRji1hXAu7K+VDphlHG1TRhuAtnrxMmucV76JSva2Wa9U8j gI5JifkjHjktTIDamTBuFEk2fySt6cR+fFeXc+5q0rTj5zRP7A+fwBa8mTFA=</latexit>apple <latexit sha1_base64="8EF/XY9hchc0JyOw3C/3J1y6s=">AB8XicbVBN S8NAEN3Ur1q/qh69LBbBU0ikqMeCF48V7Ae2oWy203bpZhN2J2IJ/RdePCji1X/jzX/jts1BWx8MPN6bYWZemEh0PO+ncLa+sbmVnG7tLO7t39QPjxqmjVHBo8lrF uh8yAFAoaKFBCO9HAolBCKxzfzPzWI2gjYnWPkwSCiA2VGAjO0EoPXYQnRMzca9c8VxvDrpK/JxUSI56r/zV7c8jUAhl8yYju8lGRMo+ASpqVuaiBhfMyG0LFUsQ hMkM0vntIzq/TpINa2FNK5+nsiY5Exkyi0nRHDkVn2ZuJ/XifFwXWQCZWkCIovFg1STGms/dpX2jgKCeWMK6FvZXyEdOMow2pZEPwl19eJc0L1790q3fVSq2ax1EkJ +SUnBOfXJEauSV10iCcKPJMXsmbY5wX5935WLQWnHzmPyB8/kD5gWRCA=</latexit>. <latexit sha1_base64="EOeu2+1jkOJLCbXnLVm9xTI40Nw=">ACBXicbVC7 SgNBFJ31GeNr1VKLwSBYhV0JahmwEasI5gHJEmYnN8mQ2dl5q4YljQ2/oqNhSK2/oOdf+PkUWjigcs9nHMvM/eEiRQGPe/bWVpeWV1bz23kN7e2d3bdvf2aiVPNocp jGetGyAxIoaCKAiU0Eg0sCiXUw8HV2K/fgzYiVnc4TCIWE+JruAMrdR2j1oID4iY3cR9RkCZbYliQRaHLXdglf0JqCLxJ+RApmh0na/Wp2YpxEo5JIZ0/S9BIOMaR RcwijfSg0kjA9YD5qWKhaBCbLJFSN6YpUO7cbalkI6UX9vZCwyZhiFdjJi2Dfz3lj8z2um2L0MqGSFEHx6UPdVFKM6TgS2hEaOMqhJYxrYf9KeZ9pxtEGl7ch+PMnL 5LaWdE/L5ZuS4VyaRZHjhySY3JKfHJByuSaVEiVcPJInskreXOenBfn3fmYji45s50D8gfO5w/S/pgX</latexit>John ate an apple . <latexit sha1_base64="Pp80LTuZSeMlwtvAyWw1qdNqkfk=">ACBXicbVA9 SwNBEN3z2/gVtdRiMQhW4U6CWgo2YhXBxEASwtxmkizu7R27c2I4YmHjX7GxUMTW/2Dnv3ETU2jig4HezPMzAsTJS35/pc3Mzs3v7C4tJxbWV1b38hvblVtnBqBFRG r2NRCsKikxgpJUlhLDEIUKrwOb86G/vUtGitjfUX9BJsRdLXsSAHkpFZ+t0F4R0QZJIlCfhH3NC/eg+ZAOGjlC37RH4FPk2BMCmyMciv/2WjHIo1Qk1BgbT3wE2pmYE gKhYNcI7WYgLiBLtYd1RChbWajLwZ83ylt3omNK018pP6eyCyth+FrjMC6tlJbyj+59VT6pw0M6mTlFCLn0WdVHGK+TAS3pYGBam+IyCMdLdy0QMDglxwORdCMPnyN KkeFoOjYumyVDgtjeNYjtsjx2wgB2zU3bOyqzCBHtgT+yFvXqP3rP35r3/tM545lt9gfexzdj3Zh1</latexit>apple John . an ate <latexit sha1_base64="UAhdBbiCS2AWJH4PQFOj5YOsA+U=">ACGnicbVDL SsNAFJ34rPFV69JNsAhuLImIuqy4cVnBtkIbymRyq0MnkzBzI8bQPxG3+h3uxK0bP8M/cNpmYVsPXDic1+cIBFco+t+WwuLS8srq6U1e31jc2u7vFNp6ThVDJosFrG 6DagGwSU0kaOA20QBjQIB7WBwOfLbD6A0j+UNZgn4Eb2TvM8ZRSP1ypUuwiOqKL9QAUdFVTbslatuzR3DmSdeQaqkQKNX/umGMUsjkMgE1brjuQn6OVXImYCh3U01J QN6B10DJU0Au3n49+HzoFRQqcfK1MSnbH6dyKnkdZFJjOiOK9nvVG4n9eJ8X+uZ9zmaQIk0O9VPhYOyMgnBCroChyAyhTHzq8PuqaIMTVxTV5APno5CSECGZlM2t G3bZOTNJjJPWsc17R2cn1SrbtFWiWyR/bJIfHIGamTK9IgTcLI3khr+TNerberQ/rc9K6YBUzu2QK1tcvnFChUQ=</latexit>Arbitrary <latexit sha1_base64="a2z5yzjTxWFv6OS4QHgkFyX408=">ACHicbVDL SsNAFJ34rPFVHzs3wSK4sSRS1KXgxmUF+4C2lMnkth06MwkzN2IN/RVxq9/hTtwKfoZ/4DTtQq0HLhzOuS9OmAhu0Pc/nYXFpeWV1cKau76xubVd3NmtmzjVDGosFrF uhtSA4ApqyFAM9FAZSigEQ6vJn7jDrThsbrFUQIdSfuK9zijaKVucb+NcI9aZlXQMsVcHXeLJb/s5/DmSTAjJTJDtVv8akcxSyUoZIa0wr8BDsZ1ciZgLHbTg0klA 1pH1qWKirBdL8+7F3ZJXI68XalkIvV39OZFQaM5Kh7ZQUB+avNxH/81op9i46GVdJiqDY9FAvFR7G3iQKL+IaGIqRJZRpbn/12IBqytAG9usK8uHDSQJqMhuGo1d1 7UZBX8TmSf103JwVq7cVEqX/iytAjkgh+SYBOScXJrUiU1wsgDeSLP5MV5dF6dN+d92rgzGb2yC84H9zEKJN</latexit>Permutation <latexit sha1_base64="MqPevcbAwFecoLBGhLWQtfp1Zg=">ACI3icbVC7 TgMxEPTx5ngFKGksIiQoiO5QBJQIGkoQhCAlUeTz7QUrPvtk7yHCKQUfg2jhO+gQDQUfwR/ghBS8RlpNLPr9U6USWExCN68sfGJyanpmVl/bn5hcam0vHJhdW41Li W2lxGzIUCmoUMJlZoClkYR61D0a+PVrMFZodY69DFop6yiRCM7QSe3SWhPhBk1aHLIO1QmtaxNbunkGuNVvl8pBJRiC/iXhiJTJCft0kcz1jxPQSGXzNpGTYKp hBwSX0/WZuIWO8yzrQcFSxFGyrGB7RpxtOiWmijSuFdKh+nyhYam0vjVxnyvDK/vYG4n9eI8dkv1UIleUIin8tSnJUdNBIjQWBjKniOMG+H+SvkVM4yjy+3HFhTd2 +0YMlCxe6nX93fZRT+TuQvudiphLuV6m1fFAdpTVD1sg62SQh2SMH5JickBrh5I48kEfy5N17z96L9/rVOuaNZlbJD3jvn6UJo8o=</latexit>Bag of Words (Set) <latexit sha1_base64="e5FYHYKt3+EkSdlUC5vA8ek/7D8=">ACLXicbVDL TgIxFO34xPE16tJNIyHBhWTGEHVJwsYlBnkQEinc4GzsO2Y8QJX+DHGLf6HS5MjFvX/oEdYCHgTdqcnHOfx404k8q2P4yV1bX1jc3Mlrm9s7u3bx0c1mUYCwo1GvJ QNF0igbMAaopDs1IAPFdDg13WE71xj0IycLgVo0i6PikH7Aeo0Rpqmvl2goelPCTcjgAYHCVf1BQAHnq3AXp+h03LWydsGeBF4Gzgxk0SwqXeun7YU09nUvyomULc eOVCchQjHKYWy2YwkRoUPSh5aGAfFBdpLJOWOc04yHe6HQTy80Yf9WJMSXcuS7OtMnaiAXtZT8T2vFqnfVSVgQxemF0G9mGMV4tQb7DEBVPGRBoQKpnfFdEAEoUo7O DdFseHjmQcRBJ7uNBqbpqk9chYdWQb184JzUSjeFLOl4sytDpGJyiPHSJSugaVANUfSEXtArejOejXfj0/iapq4Ys5ojNBfG9y/aHqia</latexit>Coherent Sentence (Sequence) <latexit sha1_base64="ujb3cYDc1gR/feni9Ij2PC0dJ1A=">ACI3icbVDL SgMxFM34dnxVXYTLIuLDNS1KWPjcsqrQptKZnMrYmSG5I9ahCz9G3Op3uBM3LvwI/8C0duHrQOBwzn3lRKkUFoPgzRsbn5icmp6Z9efmFxaXCsrZzbJDIc6T2R iLiJmQoNdRQo4SI1wFQk4TzqHg382swViS6hr0UWopdatERnKGT2oViE+EGjcprhmnbSYwCQzcOD05rm/12oRSUgyHoXxKOSImMUG0XPpxwjMFGrlk1jbCIMVWzg wKLqHvNzMLKeNdgkNRzVTYFv58BN9u6UmLoL3NIh+r3jpwpa3sqcpWK4ZX97Q3E/7xGhp29Vi50miFo/rWok0mKCR0kQmNhgKPsOcK4Ee5Wyq+YRxdbj+2oOjeb sWQgo7dpF7f932XUfg7kb/kbLsc7pQrJ5XSfmWU1gwpkjWyQUKyS/bJMamSOuHkjyQR/Lk3XvP3ov3+lU65o16VskPeO+fTr2kLA=</latexit>Transformer (BART) <latexit sha1_base64="i352KIhH/r1DVBA7M4ms/PSpKs4=">AB/XicbZDL SsNAFIYn9VbrLV52bgaL4KYlKUVdFty4rGgvUEOZTE/aoZNMnJkINRfxY0LRdz6Hu58G6dtBG39YeDjP+cw5/x+zJnSjvNl5ZaWV1bX8uFjc2t7R17d6+pRCIpNKj gQrZ9oCzCBqaQ7tWAIJfQ4tf3gxqbfuQSomohs9isELST9iAaNEG6trH1zDXQIRhZIWpR/u2kWn7EyF8HNoIgy1bv2521P0CSESFNOlOq4Tqy9lEjNKIdx4TZREB M6JH3oGIxICMpLp9uP8bFxejgQ0rxI46n7eyIloVKj0DedIdEDNV+bmP/VOokOzr2URXGizVWzj4KEYy3wJArcYxKo5iMDhEpmdsV0QCSh2gRWMCG48ycvQrNSdk/L1 atKsVbN4sijQ3SETpCLzlANXaI6aiCKHtATekGv1qP1bL1Z7PWnJXN7KM/sj6+AVPGlRs=</latexit>Sequence-to-Sequence Figure 1: Illustration of our approach for word order- ing. Bag-of-words inputs are first turned into pseudo- sequences with arbitrarily word permutations, and then fed to a sequence-to-sequence Transformer for coherent outputs. additional input. Syntactic models (Liu et al., 2015) and language models (Schmaltz et al., 2016) …

Content truncated to 500 words