Interpretable Multi-dataset Evaluation for Named Entity Recognition

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6058–6069, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6058 Interpretable Multi-dataset Evaluation for Named Entity Recognition Jinlan Fu† ∗, Pengfei Liu♯ ∗, Graham Neubig♯ † Fudan University, ♯Carnegie Mellon University fujl16@fudan.edu.cn, {pliu3,gneubig}@cs.cmu.edu Abstract With the proliferation of models for natural language processing tasks, it is even harder to understand the differences between mod- els and their relative merits. Simply looking at differences between holistic metrics such as accuracy, BLEU, or F1 does not tell us why or how particular methods perform dif- ferently and how diverse datasets influence the model design choices. In this paper, we present a general methodology for inter- pretable evaluation for the named entity recog- nition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the inter- play between them, identifying the strengths and weaknesses of current systems. By mak- ing our analysis tool available, we make it easy for future researchers to run similar anal- yses and drive progress in this area: https: //github.com/neulab/InterpretEval. 1 Introduction With improvements in model architectures (Hochre- iter and Schmidhuber, 1997; Kalchbrenner et al., 2014; Lample et al., 2016; Collobert et al., 2011) and learning of pre-trained embeddings (Peters et al., 2018; Akbik et al., 2018, 2019; Devlin et al., 2018; Pennington et al., 2014), Named Entity Recognition (NER) systems are evolving rapidly but also quickly reaching a performance plateau (Akbik et al., 2018, 2019). This proliferation of methods poses a great challenge for the current evaluation methodology, which usually is based on comparing systems on a single holistic score assess- ing accuracy (usually entity F1-score). There are several issues with this practice. First, a single eval- uation number does not allow us to distinguish on a fine-grained level the strengths and weaknesses ∗These two authors contributed equally. Life in New isYork fun . <latexit sha1_base64="hJ9fTY+N5ysrQMez0SIHtYdGc3U=">AC6X icjVHNLsVAGD3qv/4uljaNS2J109qwlNhYiJC4foJIW3OZ3N62mU6JiBewsxNbL2DLi4g34C2cGZX4iTBNv5453dO5svyhNZaN9/7nK6e3r7+gcG3a HhkdGx2vjEVpGVKhbNOEsytROFhUhkKpa6kTs5EqEnSgR21F72eS3T4UqZJZu6vNcHTC41S2ZBxqUoe1mVXZEp7rypRhTZwx7maqbZiCoVWmrts4rN X9hm+X9xMEFaijWutZ7Qn7OEKGCU6EihiROEKPjsIYCPnNwBLsgpImnzApdwqS1ZJVgRkm0zHnO3V7Ep98azsOqYf0n4Kio9zFKTsU4Rm795Nl9aZ 8P+5n1hPc3ZzvmNKq8OWY0Tsn/pPir/qzO9aLSwaHuQ7Cm3jOkurlxKeyvm5N6nrjQdcnIGHzGviGOr/Lhnz2oK27u529DmX2ylYc0+rmpLvJpTcsDB9 3H+BFvzjcBvBvz9aW5atQDmMI05jPBSxhBeto0vsK93jAo9N2rp0b5/a91OmqNJP4spy7N/8CmQk=</latexit> <latexit sha1_base64="hJ9fTY+N5ysrQMez0SIHtYdGc3U=">AC6X icjVHNLsVAGD3qv/4uljaNS2J109qwlNhYiJC4foJIW3OZ3N62mU6JiBewsxNbL2DLi4g34C2cGZX4iTBNv5453dO5svyhNZaN9/7nK6e3r7+gcG3a HhkdGx2vjEVpGVKhbNOEsytROFhUhkKpa6kTs5EqEnSgR21F72eS3T4UqZJZu6vNcHTC41S2ZBxqUoe1mVXZEp7rypRhTZwx7maqbZiCoVWmrts4rN X9hm+X9xMEFaijWutZ7Qn7OEKGCU6EihiROEKPjsIYCPnNwBLsgpImnzApdwqS1ZJVgRkm0zHnO3V7Ep98azsOqYf0n4Kio9zFKTsU4Rm795Nl9aZ 8P+5n1hPc3ZzvmNKq8OWY0Tsn/pPir/qzO9aLSwaHuQ7Cm3jOkurlxKeyvm5N6nrjQdcnIGHzGviGOr/Lhnz2oK27u529DmX2ylYc0+rmpLvJpTcsDB9 3H+BFvzjcBvBvz9aW5atQDmMI05jPBSxhBeto0vsK93jAo9N2rp0b5/a91OmqNJP4spy7N/8CmQk=</latexit> <latexit sha1_base64="hJ9fTY+N5ysrQMez0SIHtYdGc3U=">AC6X icjVHNLsVAGD3qv/4uljaNS2J109qwlNhYiJC4foJIW3OZ3N62mU6JiBewsxNbL2DLi4g34C2cGZX4iTBNv5453dO5svyhNZaN9/7nK6e3r7+gcG3a HhkdGx2vjEVpGVKhbNOEsytROFhUhkKpa6kTs5EqEnSgR21F72eS3T4UqZJZu6vNcHTC41S2ZBxqUoe1mVXZEp7rypRhTZwx7maqbZiCoVWmrts4rN X9hm+X9xMEFaijWutZ7Qn7OEKGCU6EihiROEKPjsIYCPnNwBLsgpImnzApdwqS1ZJVgRkm0zHnO3V7Ep98azsOqYf0n4Kio9zFKTsU4Rm795Nl9aZ 8P+5n1hPc3ZzvmNKq8OWY0Tsn/pPir/qzO9aLSwaHuQ7Cm3jOkurlxKeyvm5N6nrjQdcnIGHzGviGOr/Lhnz2oK27u529DmX2ylYc0+rmpLvJpTcsDB9 3H+BFvzjcBvBvz9aW5atQDmMI05jPBSxhBeto0vsK93jAo9N2rp0b5/a91OmqNJP4spy7N/8CmQk=</latexit> New York <latexit sha1_base64="rfyZ/w9Jw0WyoG2elR2YA6eAo0=">ACz3 icjVHLSsNAFD2NrxpfVZdugkXoqiTd6LgxpW0YB/SFknSaQ1NM2EysZSiuPUH3OpfiX+gf+GdMQW1iE5Icubce87MvdeLwyCRtv2aM5aWV1bX8uvmxu bW9k5hd6+Z8FT4rOHzkIu25yYsDCLWkIEMWTsWzB17IWt5o1MVb90wkQ8upDTmPXG7jAKBoHvSqK652ximeYlFyPLvCoU7bKtl7UInAwUka0aL7ygiz 4fKQYgyGCJBzCRUJPBw5sxMT1MCNOEAp0nOEWJmlTymKU4RI7ou+Qdp2MjWivPBOt9umUkF5BSgtHpOGUJwir0ywdT7WzYn/znmlPdbcp/b3Ma0ysx DWxf+nmf/VqVokBjRNQRU6wZVZ2fuaS6K+rm1peqJDnExCncp7g7GvlvM+W1iS6dtVbV8fdKZi1d7PclO8q1vSgJ2f41wEzUrZsctOvVKslrJR5 3GAQ5Ronseo4gw1NMg7xiOe8GzUjYlxZ9x/phq5TLOPb8t4+ACDcpKE</latexit> <latexit sha1_base64="rfyZ/w9Jw0WyoG2elR2YA6eAo0=">ACz3 icjVHLSsNAFD2NrxpfVZdugkXoqiTd6LgxpW0YB/SFknSaQ1NM2EysZSiuPUH3OpfiX+gf+GdMQW1iE5Icubce87MvdeLwyCRtv2aM5aWV1bX8uvmxu bW9k5hd6+Z8FT4rOHzkIu25yYsDCLWkIEMWTsWzB17IWt5o1MVb90wkQ8upDTmPXG7jAKBoHvSqK652ximeYlFyPLvCoU7bKtl7UInAwUka0aL7ygiz 4fKQYgyGCJBzCRUJPBw5sxMT1MCNOEAp0nOEWJmlTymKU4RI7ou+Qdp2MjWivPBOt9umUkF5BSgtHpOGUJwir0ywdT7WzYn/znmlPdbcp/b3Ma0ysx DWxf+nmf/VqVokBjRNQRU6wZVZ2fuaS6K+rm1peqJDnExCncp7g7GvlvM+W1iS6dtVbV8fdKZi1d7PclO8q1vSgJ2f41wEzUrZsctOvVKslrJR5 3GAQ5Ronseo4gw1NMg7xiOe8GzUjYlxZ9x/phq5TLOPb8t4+ACDcpKE</latexit> <latexit sha1_base64="rfyZ/w9Jw0WyoG2elR2YA6eAo0=">ACz3 icjVHLSsNAFD2NrxpfVZdugkXoqiTd6LgxpW0YB/SFknSaQ1NM2EysZSiuPUH3OpfiX+gf+GdMQW1iE5Icubce87MvdeLwyCRtv2aM5aWV1bX8uvmxu bW9k5hd6+Z8FT4rOHzkIu25yYsDCLWkIEMWTsWzB17IWt5o1MVb90wkQ8upDTmPXG7jAKBoHvSqK652ximeYlFyPLvCoU7bKtl7UInAwUka0aL7ygiz 4fKQYgyGCJBzCRUJPBw5sxMT1MCNOEAp0nOEWJmlTymKU4RI7ou+Qdp2MjWivPBOt9umUkF5BSgtHpOGUJwir0ywdT7WzYn/znmlPdbcp/b3Ma0ysx DWxf+nmf/VqVokBjRNQRU6wZVZ2fuaS6K+rm1peqJDnExCncp7g7GvlvM+W1iS6dtVbV8fdKZi1d7PclO8q1vSgJ2f41wEzUrZsctOvVKslrJR5 3GAQ5Ronseo4gw1NMg7xiOe8GzUjYlxZ9x/phq5TLOPb8t4+ACDcpKE</latexit> ... <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> ... <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> ... <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> ... <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> ... <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> <latexit sha1_base64="fwA7Q91L9We2Sd7+QCxG628M1/k=">ACxn icjVHLSsNAFD2Nr1pfVZdugkXoKiQi6LgpsuKthVqkWQ6rUPzYjJRShH8Abf6aeIf6F94Z0xBLaITkpw5954zc+8N0lBkynVfS9bC4tLySnm1sra+sb lV3d7pZEkuGW+zJEzkZeBnPBQxbyuhQn6ZSu5HQci7wfhUx7u3XGYiS/UJOX9yB/FYiYr4g6dxznulpzHdcsex54BaihWK2k+oIrDJCAIUcEjhiKcA gfGT09eHCREtfHlDhJSJg4xz0qpM0pi1OGT+yYviPa9Qo2pr32zIya0SkhvZKUNg5Ik1CeJKxPs08N86a/c17ajz13Sb0DwqviFiFG2L/0s0y/6vTt SgMcWJqEFRTahdHStctMVfXP7S1WKHFLiNB5QXBJmRjnrs20mald9Y38TeTqVm9Z0Vujnd9Sxqw93Oc86Bz6Hiu450d1Rr1YtRl7GEfdZrnMRpo oU2eY/wiCc8W0rtnLr7jPVKhWaXxb1sMH2haPiQ=</latexit> φeLen = 2 <latexit sha1_base64="THVoU7rek1KkclQW1AkjiLSQZXM=">AC1H icjVHLSsNAFD2Nr1ofjbp0E2wFVyXpRjdCwY0LFxXsA2wpSTpth+ZFMhFK7Urc+gNu9ZvEP9C/8M6YglpEJyQ5c+45d+be60QeT4Rpvua0peWV1bX8em Fjc2u7qO/sNpMwjV3WcEMvjNuOnTCPB6whuPBYO4qZ7TseaznjMxlv3bA4WFwJSYR6/r2MOAD7tqCqJ5eLHeiEe9N2QULZqfVck8vmRVTLWMRWBkoIV v1UH9B32EcJHCB0MAQdiDjYSea1gwERHXxZS4mBXcYZCuRNScVIYRM7pu+QdtcZG9Be5kyU26VTPHpjcho4JE9IupiwPM1Q8VRluxvuacqp7zbh P5OlsnVmBE7F+ufK/PlmLwAnqgZONUWKkdW5WZUdUXe3PhSlaAMEXES9ykeE3aVc95nQ3kSVbvsra3ib0opWbl3M2Kd3lLGrD1c5yLoFmtWGbFu qyWauVs1Hns4wBHNM9j1HCOhpq5o94wrPW1G61O+3+U6rlMs8evi3t4QNe5TR</latexit> <latexit sha1_base64="THVoU7rek1KkclQW1AkjiLSQZXM=">AC1H icjVHLSsNAFD2Nr1ofjbp0E2wFVyXpRjdCwY0LFxXsA2wpSTpth+ZFMhFK7Urc+gNu9ZvEP9C/8M6YglpEJyQ5c+45d+be60QeT4Rpvua0peWV1bX8em Fjc2u7qO/sNpMwjV3WcEMvjNuOnTCPB6whuPBYO4qZ7TseaznjMxlv3bA4WFwJSYR6/r2MOAD7tqCqJ5eLHeiEe9N2QULZqfVck8vmRVTLWMRWBkoIV v1UH9B32EcJHCB0MAQdiDjYSea1gwERHXxZS4mBXcYZCuRNScVIYRM7pu+QdtcZG9Be5kyU26VTPHpjcho4JE9IupiwPM1Q8VRluxvuacqp7zbh P5OlsnVmBE7F+ufK/PlmLwAnqgZONUWKkdW5WZUdUXe3PhSlaAMEXES9ykeE3aVc95nQ3kSVbvsra3ib0opWbl3M2Kd3lLGrD1c5yLoFmtWGbFu qyWauVs1Hns4wBHNM9j1HCOhpq5o94wrPW1G61O+3+U6rlMs8evi3t4QNe5TR</latexit> <latexit sha1_base64="THVoU7rek1KkclQW1AkjiLSQZXM=">AC1H icjVHLSsNAFD2Nr1ofjbp0E2wFVyXpRjdCwY0LFxXsA2wpSTpth+ZFMhFK7Urc+gNu9ZvEP9C/8M6YglpEJyQ5c+45d+be60QeT4Rpvua0peWV1bX8em Fjc2u7qO/sNpMwjV3WcEMvjNuOnTCPB6whuPBYO4qZ7TseaznjMxlv3bA4WFwJSYR6/r2MOAD7tqCqJ5eLHeiEe9N2QULZqfVck8vmRVTLWMRWBkoIV v1UH9B32EcJHCB0MAQdiDjYSea1gwERHXxZS4mBXcYZCuRNScVIYRM7pu+QdtcZG9Be5kyU26VTPHpjcho4JE9IupiwPM1Q8VRluxvuacqp7zbh P5OlsnVmBE7F+ufK/PlmLwAnqgZONUWKkdW5WZUdUXe3PhSlaAMEXES9ykeE3aVc95nQ3kSVbvsra3ib0opWbl3M2Kd3lLGrD1c5yLoFmtWGbFu qyWauVs1Hns4wBHNM9j1HCOhpq5o94wrPW1G61O+3+U6rlMs8evi3t4QNe5TR</latexit> 1 <latexit sha1_base64="So8BIrfS2r7ahGQeXvbRPZDZdk=">ACxn icjVHLSsNAFD2Nr1pfVZdugq3QVUm60WXBTZcV7QO0SJO69C8mEyUgR/wK1+mvgH+hfeGaegFtEJSc6ce8+Zuf6acgz6TivBWtpeWV1rbhe2tjc2t …

Content truncated to 500 words