RevUp: Revise and Update Information Bottleneck for Event Representation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 797–814 May 2-6, 2023 ©2023 Association for Computational Linguistics RevUp: Revise and Update Information Bottleneck for Event Representation Mehdi Rezaee Francis Ferraro Department of Computer Science University of Maryland Baltimore County Baltimore, MD 21250 USA {rezaee1, ferraro}@umbc.edu Abstract The existence of external (“side”) semantic knowledge has been shown to result in more expressive computational event models. To en- able the use of side information that may be noisy or missing, we propose a semi-supervised information bottleneck-based discrete latent variable model. We reparameterize the model’s discrete variables with auxiliary continuous la- tent variables and a light-weight hierarchical structure. Our model is learned to minimize the mutual information between the observed data and optional side knowledge that is not already captured by the new, auxiliary variables. We theoretically show that our approach general- izes past approaches, and perform an empiri- cal case study of our approach on event mod- eling. We corroborate our theoretical results with strong empirical experiments, showing that the proposed method outperforms previ- ous proposed approaches on multiple datasets. 1 Introduction In this work, we are interested in addressing limita- tions in how computational event modeling can make use of relevant, supplementary semantic knowledge. This is because when modeling text descriptions of complex situations, such as newspa- per descriptions of real world events, learning how to encode richer information about those descrip- tions can be a fruitful way of improving modeling performance (Judea and Strube, 2015; Xia et al., 2021). E.g., if we are dealing with sequences of events, like a newspaper report of a stock or com- merce transaction, then being able to encode that “buying” or “selling,” even though they may have nuanced connotation differences, are instances of the same general event (a TRANSACTION event) can improve downstream predictive performance on what events might happen next (Ferraro and Van Durme, 2016; Rezaee and Ferraro, 2021). However, there is not an obvious single way to learn to encode this richer information, as three dif- <latexit sha1_base64="WeBNExEPxw/eoxZuXKqkOv8u6Ts=">AB+HicbVDLSgMxFM3UV62Pjrp0 EyCqzIjoi6LblxWsQ9oh5Jb9vQJDMkmUId+iVuXCji1k9x59+YaWehrQcCh3Pu5Z6cMOZMG8/7dgpr6xubW8Xt0s7u3n7ZPThs6ihRFBo04pFqh0QDZxIahkO7VgBESGHVji+zfzWBJRmkXw0xgCQYaSDR glxko9t9wVxIyUSB9gwjTMem7Fq3pz4FXi56SCctR7le3H9FEgDSUE607vhebICXKMphVuomGmJCx2QIHUslEaCDdB58hk+t0seDSNknDZ6rvzdSIrSeitBOZjH1speJ/3mdxAyug5TJODEg6eLQIOHYRDhr AfeZAmr41BJCFbNZMR0RaixXZVsCf7yl1dJ87zqX1b9+4tK7Savo4iO0Qk6Qz6QjV0h+qogShK0DN6RW/Ok/PivDsfi9GCk+8coT9wPn8AZMeTkg=</latexit>Revise <latexit sha1_base64="5jEob8tJeJMlaCKkMFcdu5gcjxM=">AB+HicbVBNS8NAFNzUr1o/GvXoJVgETy URUY9FLx4rmLbQhrLZvLRLdzdhdyPU0F/ixYMiXv0p3vw3btoctHVgYZh5jzc7Ycqo0q7bVXW1jc2t6rbtZ3dvf26fXDYUkmCfgkYnshVgBowJ8TWDXioB85BN5zcFn73EaSiXjQ0xQCjkeCxpRgbaShXR9wrMeS534aY Q2zod1wm+4czirxStJAJdpD+2sQJSTjIDRhWKm+56Y6yLHUlDCY1QaZghSTCR5B31CBOagnwefOadGiZw4keYJ7czV3xs5kpNeWgmi5hq2SvE/7x+puPrIKcizTQIsjgUZ8zRiVO04ERUAtFsagmkpqsDhljiYk2XdVMCd7yl 1dJ57zpXTa9+4tG6aso4qO0Qk6Qx6Qi10h9rIRwRl6Bm9ojfryXqx3q2PxWjFKneO0B9Ynz9UBpOH</latexit>Update <latexit sha1_base64="Vsgicme2bHrSDsLe7T68ZUT835o=">ACRXicbVC7TsMwFHV4U14FRhaLgtQKqBIGYEFCwMAIEgWkpqpuXL e1sJPIvkGUkJ9jYWfjD1gYQIgVnNKB15UsHZ1z7sMniKUw6LqPztDwyOjY+MRkYWp6ZnauOL9wZqJEM15jkYz0RQCGSxHyGgqU/CLWHFQg+XlweZDr51dcGxGFp9iLeUNBJxRtwQAt1Sz6K3Ez9TugFGRlvL1ev6nQXep3AVNf2jEtyKj+7rD6Gi17Gz8sFZpPUYBdBjI9zPqjKivNY smtuv2if4E3ACUyqONm8cFvRSxRPEQmwZi658bYSEGjYJnBT8xPAZ2CR1etzAExU0j7aeQ0VXLtGg70vaFSPvs94UlDE9FVhnfqn5reXkf1o9wfZOIxVhnCAP2deidiIpRjSPlLaE5gxlzwJgWthbKeuCBoY2+INwfv95b/gbLPqbVW9k83S3v4gjgmyRJZJmXhkm+yRI3JMaoSR O/JEXsirc+8O2/O+5d1yBn0LJIf5Xx8AvZxsVw=</latexit> pγ(t|x, z) = ˆλrγ(t|z) + (1 − ˆλ)pD(t|x) <latexit sha1_base64="U6i0+KZTd/xRAseCc05TUk+kezs=">ACJHicbVDLSgMxFM34tr6qLt0EW6EFKTNdqOBGdCPoKthU4pmTStoUlmSO5I67Qf48ZfcePC By7c+C2mtQtPRA4nHMuN/cEkeAGXPfTmZmdm19YXFpOrayurW+kN7cqJow1ZWUailBXA2KY4IqVgYNg1UgzIgPBboLO2dC/uWPa8FBdQy9idUnairc4JWClRvo46wPrQiK5wheXAz/g7VzUSPw2kZIMctDv7t/n+39W7rPD2P5bCOdcQvuCHiaeGOSQWOUGuk3vxnSWDIFVBjap4bQT0hGjgVbJDyY8MiQjukzWqWKiKZqSejIwd4zypN3Aq1 fQrwSP09kRBpTE8GNikJ3JpJbyj+59ViaB3VE6iGJiP4tascAQ4mFjuMk1oyB6lhCquf0rprdEwq215QtwZs8eZpUigXvoOBdFTMnp+M6ltAO2kU5KFDdILOUQmVEUP6Am9oFfn0Xl23p2Pn+iM57ZRn/gfH0DOkGlJg=</latexit> min KL � pγ(t|x, z)||rγ(t|z) � <latexit sha1_base64="kwU3DeR6mv2iUKtwaBaiqc1cM7M=">AB/XicdVDLSsNAFJ34rPUVHzs3g0Wom5IUbeOuqAuXFewD2hAm0 k7dPJgZiLWGPwVNy4Ucet/uPNvnLQRVPTAwOGce7lnjhsxKqRhfGhz8wuLS8uFleLq2vrGpr613RZhzDFp4ZCFvOsiQRgNSEtSyUg34gT5LiMd3yW+Z1rwgUNgys5iYjto2FAPYqRVJKj70ZO0veRHGHEkvM0Lcu7m0NHLxkVo2YeWxZUZApFTqxatVqHZq6UQI6mo7/3ByGOfRJIz JAQPdOIpJ0gLilmJC32Y0EihMdoSHqKBsgnwk6m6VN4oJQB9EKuXiDhVP2+kSBfiInvqsksqPjtZeJfXi+WnmUnNIhiSQI8O+TFDMoQZlXAeUESzZRBGFOVaIR4gjLFVhRVXC10/h/6RdrZi1inl5VGqc5nUwB7YB2VgjpogAvQBC2AwS14AE/gWbvXHrUX7XU2OqflOzvgB7S3 TxvLlak=</latexit>pD(t|x) <latexit sha1_base64="xPSL6+QU5ZTq7JgpF9CXF69Z0u0=">AB+HicdVDLSsNAFJ3UV62PRl26GSxC3YSktNXuim5cVrAPaEOYTKft0JkzEyEN vZL3LhQxK2f4s6/cdJGUNEDFw7n3Mu9/gRo1LZ9oeRW1vf2NzKbxd2dvf2i+bBYUeGscCkjUMWip6PJGE0IG1FSO9SBDEfUa6/vQq9bt3REgaBrdqFhGXo3FARxQjpSXPLAovGYwR52hRVvfzM8s2VajZjdqNWhb9hIpqdr1agU6mVICGVqe+T4YhjmJFCYISn7jh0pN0FCUczIojCIJYkQnqIx6WsaIE6km ywPX8BTrQzhKBS6AgWX6veJBHEpZ9zXnRypifztpeJfXj9Wows3oUEUKxLg1aJRzKAKYZoCHFJBsGIzTRAWVN8K8QJhJXOqBD+PoU/k86FcupW85NtdS8zOLIg2NwAsrAegCa5BC7QBjF4AE/g2Zgbj8aL8bpqzRnZzBH4AePtEw4zk1s=</latexit>rγ(t|z) Figure 1: Revise and Update steps for an observed node. We update the proposed distribution rγ(t|z) with the empirical distribution pD(t|x) to produce the revised distribution pγ(t|x, z) (dashed). We then minimize the KL divergence between the proposed and revised distri- butions to update the proposed distribution. For unsu- pervised nodes, we rely on rγ(t|z) without updating. ferent questions naturally come to mind: (1) If the model is representationally limited, can we address these representational limitations of the model itself, such as through developing richer la- tent representations z of the input x? (2) If there is available background or side information t that may be especially relevant for the modeling task at hand, can we …

Content truncated to 500 words