{"id":7861,"date":"2024-07-02T23:01:01","date_gmt":"2024-07-02T15:01:01","guid":{"rendered":""},"modified":"2024-07-02T23:01:01","modified_gmt":"2024-07-02T15:01:01","slug":"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs","status":"publish","type":"post","link":"https:\/\/mushiming.com\/7861.html","title":{"rendered":"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs"},"content":{"rendered":"<p><svg xmlns=\"http:\/\/www.w3.org\/2000\/svg\" style=\"display: none;\"> \n <path stroke-linecap=\"round\" d=\"M5,0 0,2.5 5,5z\" id=\"raphael-marker-block\" style=\"-webkit-tap-highlight-color: rgba(0, 0, 0, 0);\"><\/path> \n<\/svg> <\/p>\n<div class=\"toc\">\n<h4>\u6587\u7ae0\u76ee\u5f55<\/h4>\n<ul>\n<li>Abstract<\/li>\n<li>1 Introduction<\/li>\n<li>2 Related Work<\/li>\n<li>3 BCNN: Basic Bi-CNN<\/li>\n<li>4 ABCNN: Attention-Based BCNN<\/li>\n<li>5 Experiments<\/li>\n<li>\n<ul>\n<li>5.1 Answer Selection<\/li>\n<li>5.2 Paraphrase Identification<\/li>\n<li>5.3 Textual Entailment<\/li>\n<\/ul>\n<\/li>\n<li>6 Summary<\/li>\n<\/ul>\n<\/div>\n<h2>Abstract<\/h2>\n<p>How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence\u2019s representation separately, rarely considering the impact of the other sentence; or (iii) relies fully on manually designed, task-specific linguistic features. This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs. (ii) We propose three attention schemes that integrate mutual influence between sentences into CNNs; thus, the representation of each sentence takes into consideration its counterpart. These interdependent sentence pair representations are more powerful than isolated sentence representations. (iii) ABCNNs achieve state-of-the-art performance on AS, PI and TE tasks. We release code at: https:\/\/github.com\/yinwenpeng\/Answer_Selection.<br \/> \u5982\u4f55\u5bf9\u4e00\u5bf9\u53e5\u5b50\u5efa\u6a21\u662f\u8bb8\u591aNLP\u4efb\u52a1\u4e2d\u7684\u5173\u952e\u95ee\u9898\uff0c\u4f8b\u5982\u7b54\u6848\u9009\u62e9\uff08AS\uff09\uff0c\u6bb5\u843d\u8bc6\u522b\uff08PI\uff09\u548c\u6587\u672c\u8574\u6db5\uff08TE\uff09\u3002\u5927\u591a\u6570\u5148\u524d\u7684\u5de5\u4f5c\uff08i\uff09\u901a\u8fc7\u5fae\u8c03\u7279\u5b9a\u7cfb\u7edf\u6765\u5904\u7406\u4e00\u9879\u5355\u72ec\u7684\u4efb\u52a1; \uff08ii\uff09\u5206\u522b\u5bf9\u6bcf\u4e2a\u53e5\u5b50\u7684\u8868\u793a\u8fdb\u884c\u5efa\u6a21\uff0c\u5f88\u5c11\u8003\u8651\u53e6\u4e00\u53e5\u8bdd\u7684\u5f71\u54cd;\u6216\uff08iii\uff09\u5b8c\u5168\u4f9d\u8d56\u4e8e\u624b\u52a8\u8bbe\u8ba1\u7684\uff0c\u4efb\u52a1\u7279\u5b9a\u7684\u8bed\u8a00\u7279\u5f81\u3002\u8fd9\u9879\u5de5\u4f5c\u63d0\u51fa\u4e86\u4e00\u822c\u7684\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff08ABCNN\uff09\uff0c\u7528\u4e8e\u5efa\u6a21\u4e00\u5bf9\u53e5\u5b50\u3002\u6211\u4eec\u505a\u4e86\u4e09\u4e2a\u8d21\u732e\u3002 \uff08i\uff09<strong>ABCNN\u53ef\u4ee5\u5e94\u7528\u4e8e\u9700\u8981\u5bf9\u53e5\u5b50\u5bf9\u8fdb\u884c\u5efa\u6a21\u7684\u5404\u79cd\u4efb\u52a1<\/strong>\u3002 \uff08ii\uff09\u6211\u4eec\u63d0\u51fa\u4e09\u79cd\u6ce8\u610f\u65b9\u6848\uff0c<strong>\u5c06\u53e5\u5b50\u4e4b\u95f4\u7684\u76f8\u4e92\u5f71\u54cd\u7eb3\u5165CNN<\/strong>;\u56e0\u6b64\uff0c\u6bcf\u4e2a\u53e5\u5b50\u7684\u8868\u793a\u90fd\u8003\u8651\u5230\u4e86\u5b83\u7684\u5bf9\u5e94\u7269\u3002\u8fd9\u4e9b\u76f8\u4e92\u4f9d\u8d56\u7684\u53e5\u5b50\u5bf9\u8868\u793a\u6bd4\u5b64\u7acb\u7684\u53e5\u5b50\u8868\u793a\u66f4\u5f3a\u5927\u3002 \uff08iii\uff09ABCNN\u5728AS\uff0cPI\u548cTE\u4efb\u52a1\u4e0a\u5b9e\u73b0<strong>\u6700\u5148\u8fdb\u7684\u6027\u80fd<\/strong>\u3002\u6211\u4eec\u5728\u4ee5\u4e0b\u7f51\u5740\u53d1\u5e03\u4ee3\u7801\uff1ahttps\uff1a\/\/github.com\/yinwenpeng\/Answer_Selection\u3002<\/p>\n<h2>1 Introduction<\/h2>\n<p>How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS) (Yu et al., 2014; Feng et al., 2015), paraphrase identification (PI) (Madnani et al., 2012; Yin and Schutze, 2015a), textual entailment (TE) (Marelli et al., 2014a; Bowman et al., 2015a) etc.<br \/> \u5728\u8bb8\u591aNLP\u4efb\u52a1\u4e2d\uff0c\u5982\u4f55\u4e3a\u4e00\u5bf9\u53e5\u5b50\u5efa\u6a21\u662f\u4e00\u4e2a\u5173\u952e\u95ee\u9898\uff0c\u5982\u7b54\u6848\u9009\u62e9\uff08AS\uff09\uff08Yu\u7b49\u4eba\uff0c2014; Feng\u7b49\u4eba\uff0c2015\uff09\uff0c\u6bb5\u843d\u8bc6\u522b\uff08PI\uff09\uff08Madnani\u7b49\uff0c2012 ; Yin\u548cSchutze\uff0c2015a\uff09\uff0c\u6587\u672c\u8574\u542b\uff08TE\uff09\uff08Marelli\u7b49\uff0c2014a; Bowman\u7b49\uff0c2015a\uff09\u7b49\u3002<br \/> <img decoding=\"async\" src=\"https:\/\/img.mushiming.top\/app\/mushiming_com\/2d960f3e3dea44c6afaf6fb969d4374c.png\" =\"\" =\"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\" alt=\"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (https:\/\/mushiming.com\/)  \u7b2c1\u5f20\" title=\"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs  \u7b2c1\u5f20-\u7a46\u4e16\u660e\u535a\u5ba2\" ><\/p>\n<p>Most prior work derives each sentence\u2019s representation separately, rarely considering the impact of the other sentence. This neglects the mutual influence of the two sentences in the context of the task. It also contradicts what humans do when comparing two sentences. We usually focus on key parts of one sentence by extracting parts from the other sentence that are related by identity, synonymy, antonymy and other relations. Thus, human beings model the two sentences together, using the content of one sentence to guide the representation of the other.<br \/> <font color=\"red\">\u5927\u591a\u6570\u5148\u524d\u7684\u5de5\u4f5c\u5206\u522b\u5f97\u51fa\u6bcf\u4e2a\u53e5\u5b50\u7684\u8868\u793a\uff0c\u5f88\u5c11\u8003\u8651\u53e6\u4e00\u4e2a\u53e5\u5b50\u7684\u5f71\u54cd\u3002 \u8fd9\u5ffd\u7565\u4e86\u4e24\u4e2a\u53e5\u5b50\u5728\u4efb\u52a1\u80cc\u666f\u4e0b\u7684\u76f8\u4e92\u5f71\u54cd<\/font>\u3002 \u5b83\u4e5f\u4e0e\u4eba\u7c7b\u5728\u6bd4\u8f83\u4e24\u4e2a\u53e5\u5b50\u65f6\u7684\u884c\u4e3a\u76f8\u77db\u76fe\u3002 \u6211\u4eec\u901a\u5e38\u901a\u8fc7\u4ece\u53e6\u4e00\u4e2a\u53e5\u5b50\u4e2d\u63d0\u53d6\u4e0e\u8eab\u4efd\uff0c\u540c\u4e49\u8bcd\uff0c\u53cd\u4e49\u548c\u5176\u4ed6\u5173\u7cfb\u76f8\u5173\u7684\u90e8\u5206\u6765\u5173\u6ce8\u4e00\u4e2a\u53e5\u5b50\u7684\u5173\u952e\u90e8\u5206\u3002 \u56e0\u6b64\uff0c\u4eba\u7c7b\u5c06\u4e24\u4e2a\u53e5\u5b50\u4e00\u8d77\u5efa\u6a21\uff0c\u4f7f\u7528\u4e00\u4e2a\u53e5\u5b50\u7684\u5185\u5bb9\u6765\u6307\u5bfc\u53e6\u4e00\u4e2a\u53e5\u5b50\u7684\u8868\u793a\u3002<\/p>\n<p>Figure 1 demonstrates that each sentence of a pair partially determines which parts of the other sentence we must focus on. For AS, correctly answering <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 0 s_0 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> requires attention on \u201cgross\u201d: <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 + s^+_ 1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> contains a corresponding unit (\u201cearned\u201d) while <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 \u2212 s^\u2212 _1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> does not. For PI, focus should be removed from \u201ctoday\u201d to correctly recognize <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 + &gt; &lt; s_0, s^+_ 1 &gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span> as paraphrases and <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 \u2212 &gt; &lt; s_0, s^\u2212 _1 &gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span> as non-paraphrases. For TE, we need to focus on \u201cfull of people\u201d (to recognize TE for <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 + &gt; &lt;s_0, s^+ _1 &gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>) and on \u201coutdoors\u201d \/ \u201cindoors\u201d (to recognize non-TE for <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 \u2212 &gt; &lt; s_0, s^\u2212 _1 &gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>). These examples show the need for an architecture that computes different representations of <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s i s_i <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.311664em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathdefault mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> for different <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 \u2212 i ( i \u2208 { 0 , 1 } ) s_{1\u2212i} (i \\in \\{0,1\\}) <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.311664em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\">1<\/span><span class=\"mord mtight\">\u2212<\/span><span class=\"mord mathdefault mtight\">i<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.208331em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mopen\">(<\/span><span class=\"mord mathdefault\">i<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">\u2208<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mopen\">{<br \/>\n      <!-- --><\/span><span class=\"mord\">0<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\">1<\/span><span class=\"mclose\">}<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span>.<br \/> \u56fe1\u8868\u660e\uff0c<font color=\"blue\">\u4e00\u5bf9\u4e2d\u7684\u6bcf\u4e2a\u53e5\u5b50\u90e8\u5206\u5730\u51b3\u5b9a\u4e86\u6211\u4eec\u5fc5\u987b\u5173\u6ce8\u7684\u53e6\u4e00\u4e2a\u53e5\u5b50\u7684\u54ea\u4e9b\u90e8\u5206<\/font>\u3002 \u5bf9\u4e8eAS\uff0c\u6b63\u786e\u56de\u7b54<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 0 s_0 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u9700\u8981\u6ce8\u610f\u201c\u7c97\u7565\u201d\uff1a<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 + s^ +_ 1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u5305\u542b\u76f8\u5e94\u7684\u5355\u4f4d\uff08\u201c\u8d62\u5f97\u201d\uff09\u800c<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 \u2212 s^-_1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u4e0d\u5305\u542b\u3002 \u5bf9\u4e8ePI\uff0c\u5e94\u4ece\u201c\u4eca\u5929\u201d\u4e2d\u5220\u9664\u7126\u70b9\u4ee5\u6b63\u786e\u8bc6\u522b<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 + &gt; &lt;s0, s^ +_ 1&gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord mathdefault\">s<\/span><span class=\"mord\">0<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>\u4f5c\u4e3a\u91ca\u4e49\u548c<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 \u2212 &gt; &lt;s_0, s^-_1&gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>\u4f5c\u4e3a\u975e\u91ca\u4e49\u3002 \u5bf9\u4e8eTE\uff0c\u6211\u4eec\u9700\u8981\u5173\u6ce8\u201c\u6ee1\u6ee1\u7684\u4eba\u201d\uff08\u8bc6\u522bTE\u4e3a<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 + &gt; &lt;s0, s^ +_ 1&gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord mathdefault\">s<\/span><span class=\"mord\">0<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">+<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>\uff09\u548c\u201c\u6237\u5916\u201d\/\u201c\u5ba4\u5185\u201d\uff08\u4ee5\u8bc6\u522b\u975eTE\u4e3a<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> &lt; s 0 , s 1 \u2212 &gt; &lt;s_0, s^-_1&gt; <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.5782em; vertical-align: -0.0391em;\"><\/span><span class=\"mrel\">&lt;<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1.07777em; vertical-align: -0.266308em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.811462em;\"><span class=\"\" style=\"top: -2.43369em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><span class=\"\" style=\"top: -3.10313em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mbin mtight\">\u2212<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.266308em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">&gt;<\/span><\/span><\/span><\/span><\/span>\uff09\u3002<font color=\"red\"> \u8fd9\u4e9b\u793a\u4f8b\u8868\u660e\u9700\u8981\u4e00\u79cd\u4f53\u7cfb\u7ed3\u6784\uff0c\u8be5\u4f53\u7cfb\u7ed3\u6784\u4e3a\u4e0d\u540c\u7684<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 \u2212 i ( i \u2208 { 0 , 1 } ) s_{1\u2212i} (i \\in \\{0,1\\}) <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.311664em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\"><span class=\"mord mtight\">1<\/span><span class=\"mord mtight\">\u2212<\/span><span class=\"mord mathdefault mtight\">i<\/span><\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.208331em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mopen\">(<\/span><span class=\"mord mathdefault\">i<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">\u2208<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mopen\">{<br \/>\n       <!-- --><\/span><span class=\"mord\">0<\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\">1<\/span><span class=\"mclose\">}<\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span>\u8ba1\u7b97<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s i s_i <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.311664em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mathdefault mtight\">i<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u7684\u4e0d\u540c\u8868\u793a\u3002<\/font><\/p>\n<p>Convolutional Neural Networks (CNNs) (LeCun et al., 1998) are widely used to model sentences (Kalchbrenner et al., 2014; Kim, 2014) and sentence pairs (Socher et al., 2011; Yin and Schutze, 2015a), especially in classification tasks. CNNs are supposed to be good at extracting robust and abstract features of input. <strong>This work presents the ABCNN, an attention-based convolutional neural network, that has a powerful mechanism for modeling a sentence pair by taking into account the interdependence between the two sentences.<\/strong> The ABCNN is a general architecture that can handle a wide variety of sentence pair modeling tasks.<br \/> \u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff08CNNs\uff09\uff08LeCun\u7b49\uff0c1998\uff09\u88ab\u5e7f\u6cdb\u7528\u4e8e\u53e5\u5b50\u6a21\u578b\uff08Kalchbrenner\u7b49\uff0c2014; Kim\uff0c2014\uff09\u548c\u53e5\u5b50\u5bf9\uff08Socher\u7b49\uff0c2011; Yin\u548cSchutze\uff0c2015a\uff09\uff0c\u7279\u522b\u662f\u5728\u5206\u7c7b\u4efb\u52a1\u4e2d\u3002 CNN\u5584\u4e8e\u63d0\u53d6\u8f93\u5165\u7684\u5f3a\u5927\u548c\u62bd\u8c61\u7684\u7279\u5f81\u3002<strong>\u8fd9\u9879\u5de5\u4f5c\u63d0\u51fa\u4e86ABCNN\uff0c\u4e00\u79cd\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u5377\u79ef\u795e\u7ecf\u7f51\u7edc\uff0c\u5b83\u5177\u6709\u5f3a\u5927\u7684\u673a\u5236\uff0c\u901a\u8fc7\u8003\u8651\u4e24\u4e2a\u53e5\u5b50\u4e4b\u95f4\u7684\u76f8\u4e92\u4f9d\u8d56\u6027\u6765\u5efa\u6a21\u53e5\u5b50\u5bf9\u3002<\/strong> ABCNN\u662f\u4e00\u79cd\u901a\u7528\u67b6\u6784\uff0c\u53ef\u4ee5\u5904\u7406\u5404\u79cd\u53e5\u5b50\u5bf9\u5efa\u6a21\u4efb\u52a1\u3002<\/p>\n<p>Some prior work proposes simple mechanisms that can be interpreted as controlling varying attention; e.g., Yih et al. (2013) employ word alignment to match related parts of the two sentences. In contrast, our attention scheme based on CNNs models relatedness between two parts fully automatically. <strong>Moreover, attention at multiple levels of granularity, not only at word level, is achieved as we stack multiple convolution layers that increase abstraction.<\/strong><br \/> \u4e00\u4e9b\u5148\u524d\u7684\u5de5\u4f5c\u63d0\u51fa\u4e86\u7b80\u5355\u7684\u673a\u5236\uff0c\u53ef\u4ee5\u89e3\u91ca\u4e3a\u63a7\u5236\u4e0d\u540c\u7684\u6ce8\u610f\u529b;\u4f8b\u5982\uff0cYih\u7b49\u4eba\u3002 \uff082013\uff09\u4f7f\u7528\u5355\u8bcd\u5bf9\u9f50\u6765\u5339\u914d\u4e24\u4e2a\u53e5\u5b50\u7684\u76f8\u5173\u90e8\u5206\u3002\u76f8\u6bd4\u4e4b\u4e0b\uff0c\u6211\u4eec\u57fa\u4e8eCNN\u7684\u5173\u6ce8\u65b9\u6848\u5b8c\u5168\u81ea\u52a8\u5730\u6a21\u62df\u4e86\u4e24\u4e2a\u90e8\u5206\u4e4b\u95f4\u7684\u76f8\u5173\u6027\u3002\u6b64\u5916\uff0c<font color=\"red\">\u5f53\u6211\u4eec\u5806\u53e0\u591a\u4e2a\u589e\u52a0\u62bd\u8c61\u7684\u5377\u79ef\u5c42\u65f6\uff0c\u4e0d\u4ec5\u5728\u8bcd\u7ea7\u4e0a\u5b9e\u73b0\u4e86\u5bf9\u591a\u4e2a\u7c92\u5ea6\u7ea7\u522b\u7684\u5173\u6ce8\u3002<\/font><\/p>\n<p>Prior work on attention in deep learning (DL) mostly addresses long short-term memory networks (LSTMs) (Hochreiter and Schmidhuber, 1997). LSTMs achieve attention usually in a word-to-word scheme, and word representations mostly encode the whole context within the sentence (Bahdanau et al., 2015; Rocktaschel et al., 2016). It is not clear whether this is the best strategy; e.g., in the AS example in Figure 1, it is possible to determine that \u201chow much\u201d in <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 0 s_0 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> matches \u201c$161.5 million\u201d in <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 s_1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span> without taking the entire sentence contexts into account. This observation was also investigated by Yao et al. (2013b) where an information retrieval system retrieves sentences with tokens labeled as DATE by named entity recognition or as CD by POS tagging if there is a \u201cwhen\u201d question. However, labels or POS tags require extra tools. CNNs benefit from incorporating attention into representations of local phrases detected by filters; in contrast, LSTMs encode the whole context to form attention-based word representations \u2013 a strategy that is more complex than the CNN strategy and (as our experiments suggest) performs less well for some tasks.<br \/> \u6df1\u5ea6\u5b66\u4e60\uff08DL\uff09\u4e2d\u5173\u6ce8\u7684\u5148\u524d\u5de5\u4f5c\u4e3b\u8981\u6d89\u53ca\u957f\u671f\u77ed\u671f\u8bb0\u5fc6\u7f51\u7edc\uff08LSTM\uff09\uff08Hochreiter\u548cSchmidhuber\uff0c1997\uff09\u3002 LSTM\u901a\u5e38\u5728\u5355\u8bcd\u5230\u5355\u8bcd\u65b9\u6848\u4e2d\u83b7\u5f97\u5173\u6ce8\uff0c\u5e76\u4e14\u5355\u8bcd\u8868\u793a\u4e3b\u8981\u7f16\u7801\u53e5\u5b50\u4e2d\u7684\u6574\u4e2a\u4e0a\u4e0b\u6587\uff08Bahdanau\u7b49\uff0c2015; Rocktaschel\u7b49\uff0c2016\uff09\u3002\u76ee\u524d\u5c1a\u4e0d\u6e05\u695a\u8fd9\u662f\u5426\u662f\u6700\u4f73\u7b56\u7565;\u4f8b\u5982\uff0c\u5728\u56fe1\u7684AS\u793a\u4f8b\u4e2d\uff0c\u53ef\u4ee5\u786e\u5b9a<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 0 s_0 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u4e2d\u7684\u201c\u591a\u5c11\u201d\u4e0e<span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s 1 s_1 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.58056em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>\u4e2d\u7684\u201c$ 161.5\u767e\u4e07\u201d\u5339\u914d\uff0c\u800c\u4e0d\u8003\u8651\u6574\u4e2a\u53e5\u5b50\u4e0a\u4e0b\u6587\u3002 Yao\u7b49\u4eba\uff082013b\uff09\u4e5f\u7814\u7a76\u4e86\u8fd9\u4e00\u89c2\u5bdf\u7ed3\u679c\uff0c \u5176\u4e2d\u4fe1\u606f\u68c0\u7d22\u7cfb\u7edf\u901a\u8fc7\u547d\u540d\u5b9e\u4f53\u8bc6\u522b\u6765\u68c0\u7d22\u5177\u6709\u6807\u8bb0\u4e3aDATE\u7684\u6807\u8bb0\u7684\u53e5\u5b50\uff0c\u6216\u8005\u5982\u679c\u5b58\u5728\u201c\u4f55\u65f6\u201d\u95ee\u9898\u5219\u901a\u8fc7POS\u6807\u8bb0\u68c0\u7d22CD\u3002\u4f46\u662f\uff0c\u6807\u7b7e\u6216POS\u6807\u7b7e\u9700\u8981\u989d\u5916\u7684\u5de5\u5177\u3002 CNN\u53d7\u76ca\u4e8e\u5c06\u6ce8\u610f\u529b\u96c6\u4e2d\u5230\u8fc7\u6ee4\u5668\u68c0\u6d4b\u5230\u7684\u672c\u5730\u77ed\u8bed\u7684\u8868\u793a\u4e2d;\u76f8\u53cd\uff0cLSTM\u5bf9\u6574\u4e2a\u4e0a\u4e0b\u6587\u8fdb\u884c\u7f16\u7801\u4ee5\u5f62\u6210\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u5355\u8bcd\u8868\u793a - \u8fd9\u79cd\u7b56\u7565\u6bd4CNN\u7b56\u7565\u66f4\u590d\u6742\uff0c\u5e76\u4e14\uff08\u5982\u6211\u4eec\u7684\u5b9e\u9a8c\u6240\u793a\uff09\u5bf9\u67d0\u4e9b\u4efb\u52a1\u6267\u884c\u5f97\u4e0d\u592a\u597d\u3002<\/p>\n<p>Apart from these differences, it is clear that attention has as much potential for CNNs as it does for LSTMs. As far as we know, this is the first NLP paper that incorporates attention into CNNs. Our ABCNNs get state-of-the-art in AS and TE tasks, and competitive performance in PI, then obtains further improvements over all three tasks when linguistic features are used.<br \/> \u9664\u4e86\u8fd9\u4e9b\u5dee\u5f02\u4e4b\u5916\uff0c\u5f88\u660e\u663e\uff0c\u6ce8\u610f\u529b\u4e0eCNS\u76f8\u6bd4\u5177\u6709\u4e0eLSTM\u4e00\u6837\u591a\u7684\u6f5c\u529b\u3002\u636e\u6211\u4eec\u6240\u77e5\uff0c\u8fd9\u662f\u7b2c\u4e00\u4efd\u5c06\u6ce8\u610f\u529b\u7eb3\u5165CNN\u7684NLP\u8bba\u6587\u3002\u6211\u4eec\u7684ABCNN\u5728AS\u548cTE\u4efb\u52a1\u4e2d\u83b7\u5f97\u6700\u5148\u8fdb\u7684\u6280\u672f\uff0c\u5728PI\u4e2d\u5177\u6709\u7ade\u4e89\u6027\uff0c\u7136\u540e\u5728\u4f7f\u7528\u8bed\u8a00\u7279\u5f81\u65f6\u83b7\u5f97\u5bf9\u6240\u6709\u4e09\u4e2a\u4efb\u52a1\u7684\u8fdb\u4e00\u6b65\u6539\u8fdb\u3002<\/p>\n<h2>2 Related Work<\/h2>\n<p><strong>Non-DL on Sentence Pair Modeling.<\/strong> Sentence pair modeling has attracted lots of attention in the past decades. Many tasks can be reduced to a semantic text matching problem. In this paper, we adopt the arguments by Yih et al. (2013) who argue against shallow approaches as well as against semantic text matching approaches that can be computationally expensive:<br \/> <strong>\u53e5\u5b50\u5bf9\u5efa\u6a21\u7684\u975eDL\u65b9\u6cd5\u3002<\/strong><br \/> \u5728\u8fc7\u53bb\u7684\u51e0\u5341\u5e74\u4e2d\uff0c\u53e5\u5b50\u5bf9\u5efa\u6a21\u5f15\u8d77\u4e86\u5f88\u591a\u5173\u6ce8\u3002 \u8bb8\u591a\u4efb\u52a1\u53ef\u4ee5\u7b80\u5316\u4e3a\u8bed\u4e49\u6587\u672c\u5339\u914d\u95ee\u9898\u3002 \u5728\u672c\u6587\u4e2d\uff0c\u6211\u4eec\u91c7\u7528\u4e86Yih\u7b49\u4eba\uff082013\uff09\u7684\u8bba\u8bc1\uff0c\u4ed6\u4eec\u53cd\u5bf9\u6d45\u5c42\u65b9\u6cd5\u4ee5\u53ca\u8bed\u4e49\u6587\u672c\u5339\u914d\u65b9\u6cd5\uff0c\u8fd9\u4e9b\u65b9\u6cd5\u8ba1\u7b97\u6210\u672c\u5f88\u9ad8\uff1a<\/p>\n<p>Due to the variety of word choices and inherent ambiguities in natural language, bag-of-word approaches with simple surface-form word matching tend to produce brittle results with poor prediction accuracy (Bilotti et al., 2007). As a result, researchers put more emphasis on exploiting syntactic and semantic structure. Representative examples include methods based on deeper semantic analysis (Shen and Lapata, 2007; Moldovan et al., 2007), tree edit-distance (Punyakanok et al., 2004; Heilman and Smith, 2010) and quasi-synchronous grammars (Wang et al., 2007) that match the dependency parse trees of the two sentences.<br \/> \u7531\u4e8e\u81ea\u7136\u8bed\u8a00\u4e2d\u8bcd\u8bed\u9009\u62e9\u7684\u591a\u6837\u6027\u548c\u56fa\u6709\u7684\u6a21\u7cca\u6027\uff0c\u5177\u6709\u7b80\u5355\u8868\u9762\u5f62\u5f0f\u8bcd\u5339\u914d\u7684\u8bcd\u888b\u65b9\u6cd5\u5f80\u5f80\u4f1a\u4ea7\u751f\u8106\u5f31\u7684\u7ed3\u679c\uff0c\u9884\u6d4b\u51c6\u786e\u6027\u8f83\u5dee\uff08Bilotti\u7b49\uff0c2007\uff09\u3002 \u56e0\u6b64\uff0c\u7814\u7a76\u4eba\u5458\u66f4\u52a0\u91cd\u89c6\u5229\u7528\u53e5\u6cd5\u548c\u8bed\u4e49\u7ed3\u6784\u3002 \u4ee3\u8868\u6027\u7684\u4f8b\u5b50\u5305\u62ec\u57fa\u4e8e<font color=\"red\">\u6df1\u5ea6\u8bed\u4e49\u5206\u6790\u7684\u65b9\u6cd5<\/font>\uff08Shen\u548cLapata\uff0c2007; Moldovan\u7b49\uff0c2007\uff09\uff0c\u6811\u7f16\u8f91\u8ddd\u79bb\uff08Punyakanok\u7b49\uff0c2004; Heilman\u548cSmith\uff0c2010\uff09\u548c\u51c6\u540c\u6b65\u8bed\u6cd5\uff08Wang et al\uff0c2007\uff09\u5339\u914d\u4e24\u4e2a\u53e5\u5b50\u7684\u4f9d\u8d56\u89e3\u6790\u6811\u3002<\/p>\n<p>Instead of focusing on the high-level semantic representation, Yih et al. (2013) turn their attention to improving the shallow semantic component, lexical semantics, by performing semantic matching based on a latent word-alignment structure (cf. Chang et al. (2010)). Lai and Hockenmaier (2014) explore finergrained word overlap and alignment between two sentences using negation, hypernym, synonym and antonym relations. Yao et al. (2013a) extend word-to-word alignment to phrase-to-phrase alignment by a semi-Markov CRF. However, such approaches often require more computational resources. In addition, employing syntactic or semantic parsers \u2013 which produce errors on many sentences \u2013 to find the best match between the structured representations of two sentences is not trivial.<br \/> \u800c\u4e0d\u662f\u4e13\u6ce8\u4e8e\u9ad8\u7ea7\u8bed\u4e49\u8868\u793a\uff0cYih\u7b49\uff082013\uff09\u901a\u8fc7\u57fa\u4e8e\u6f5c\u5728\u7684\u8bcd\u5bf9\u9f50\u7ed3\u6784\u6267\u884c\u8bed\u4e49\u5339\u914d\uff08\u53c2\u89c1Chang\u7b49\u4eba\uff082010\uff09\uff09\uff0c\u5c06\u6ce8\u610f\u529b\u8f6c\u5411\u6539\u8fdb\u6d45\u5c42\u8bed\u4e49\u6210\u5206\uff0c\u8bcd\u6c47\u8bed\u4e49\u3002 Lai\u548cHockenmaier\uff082014\uff09\u4f7f\u7528\u5426\u5b9a\uff0c\u4e0a\u4f4d\u8bcd\uff0c\u540c\u4e49\u8bcd\u548c\u53cd\u4e49\u8bcd\u5173\u7cfb\u6765\u63a2\u7d22\u4e24\u4e2a\u53e5\u5b50\u4e4b\u95f4\u7684\u7ec6\u7c92\u5ea6\u91cd\u53e0\u548c\u5bf9\u9f50\u3002 \u59da\u7b49\u4eba\uff082013a\uff09\u901a\u8fc7\u534a\u9a6c\u5c14\u53ef\u592bCRF\u5c06\u8bcd\u5bf9\u9f50\u6269\u5c55\u5230\u8bcd\u7ec4\u5230\u8bcd\u7ec4\u7684\u5bf9\u9f50\u3002 \u7136\u800c\uff0c\u8fd9\u79cd\u65b9\u6cd5\u901a\u5e38\u9700\u8981\u66f4\u591a\u7684\u8ba1\u7b97\u8d44\u6e90\u3002 \u6b64\u5916\uff0c\u4f7f\u7528\u53e5\u6cd5\u6216\u8bed\u4e49\u89e3\u6790\u5668\uff08\u5728\u8bb8\u591a\u53e5\u5b50\u4e0a\u4ea7\u751f\u9519\u8bef\uff09\u6765\u627e\u5230\u4e24\u4e2a\u53e5\u5b50\u7684\u7ed3\u6784\u5316\u8868\u793a\u4e4b\u95f4\u7684\u6700\u4f73\u5339\u914d\u5e76\u975e\u6613\u4e8b\u3002<\/p>\n<p><strong>DL on Sentence Pair Modeling.<\/strong><br \/> To address some of the challenges of non-DL work, much recent work uses neural networks to model sentence pairs for AS, PI and TE.<br \/> <strong>\u5173\u4e8e\u53e5\u5b50\u5bf9\u5efa\u6a21\u7684DL\u3002<\/strong><br \/> \u4e3a\u4e86\u89e3\u51b3\u975eDL\u5de5\u4f5c\u7684\u4e00\u4e9b\u6311\u6218\uff0c\u6700\u8fd1\u7684\u5de5\u4f5c\u4f7f\u7528\u795e\u7ecf\u7f51\u7edc\u6765\u6a21\u62dfAS\uff0cPI\u548cTE\u7684\u53e5\u5b50\u5bf9\u3002<\/p>\n<p>For AS, Yu et al. (2014) present a bigram CNN to model question and answer candidates. Yang et al. (2015) extend this method and get state-of-the-art performance on the WikiQA dataset (Section 5.1). Feng et al. (2015) test various setups of a bi-CNN architecture on an insurance domain QA dataset. Tan et al. (2016) explore bidirectional LSTMs on the same dataset. <strong>Our approach is different because we do not model the sentences by two independent neural networks in parallel, but instead as an interdependent sentence pair, using attention.<\/strong><br \/> \u5bf9\u4e8eAS\uff0cYu\u7b49\u4eba\uff082014\uff09\u63d0\u51fa\u4e00\u4e2a\u4e8c\u5143\u7ec4CNN\u6765\u6a21\u62df\u95ee\u7b54\u5019\u9009\u4eba\u3002 \u6768\u7b49\u4eba\uff082015\uff09\u6269\u5c55\u6b64\u65b9\u6cd5\u5e76\u5728WikiQA\u6570\u636e\u96c6\u4e0a\u83b7\u5f97\u6700\u5148\u8fdb\u7684\u6027\u80fd\uff08\u7b2c5.1\u8282\uff09\u3002 \u51af\u7b49\u4eba\uff082015\uff09\u5728\u4fdd\u9669\u9886\u57dfQA\u6570\u636e\u96c6\u4e0a\u6d4b\u8bd5\u53ccCNN\u67b6\u6784\u7684\u5404\u79cd\u8bbe\u7f6e\u3002 Tan\u7b49\u4eba\uff082016\uff09\u5728\u540c\u4e00\u6570\u636e\u96c6\u4e0a\u63a2\u7d22\u53cc\u5411LSTM\u3002 <strong>\u6211\u4eec\u7684\u65b9\u6cd5\u662f\u4e0d\u540c\u7684\uff0c\u56e0\u4e3a\u6211\u4eec\u4e0d\u662f\u901a\u8fc7\u4e24\u4e2a\u72ec\u7acb\u7684\u795e\u7ecf\u7f51\u7edc\u5e76\u884c\u5efa\u7acb\u53e5\u5b50\uff0c\u800c\u662f\u4f5c\u4e3a\u4e00\u4e2a\u76f8\u4e92\u4f9d\u8d56\u7684\u53e5\u5b50\u5bf9\uff0c\u4f7f\u7528\u6ce8\u610f\u3002<\/strong><\/p>\n<p>For PI, Blacoe and Lapata (2012) form sentence representations by summing up word embeddings. Socher et al. (2011) use recursive autoencoders (RAEs) to model representations of local phrases in sentences, then pool similarity values of phrases from the two sentences as features for binary classification. Yin and Schutze (2015a) similarly replace an RAE with a CNN. <font color=\"red\"> In all three papers, the representation of one sentence is not influenced by the other<\/font> \u2013 in contrast to our attention-based model.<br \/> \u5bf9\u4e8ePI\uff0cBlacoe\u548cLapata\uff082012\uff09\u901a\u8fc7\u603b\u7ed3\u5355\u8bcd\u5d4c\u5165\u6765\u5f62\u6210\u53e5\u5b50\u8868\u793a\u3002 Socher\u7b49\u3002 \uff082011\uff09\u4f7f\u7528\u9012\u5f52\u81ea\u52a8\u7f16\u7801\u5668\uff08RAE\uff09\u6765\u6a21\u62df\u53e5\u5b50\u4e2d\u7684\u5c40\u90e8\u77ed\u8bed\u7684\u8868\u793a\uff0c\u7136\u540e\u5c06\u6765\u81ea\u4e24\u4e2a\u53e5\u5b50\u7684\u77ed\u8bed\u7684\u76f8\u4f3c\u5ea6\u503c\u4f5c\u4e3a\u4e8c\u5143\u5206\u7c7b\u7684\u7279\u5f81\u3002 Yin\u548cSchutze\uff082015a\uff09\u540c\u6837\u7528CNN\u53d6\u4ee3RAE\u3002 <font color=\"red\">\u5728\u6240\u6709\u4e09\u7bc7\u8bba\u6587\u4e2d\uff0c\u4e00\u4e2a\u53e5\u5b50\u7684\u8868\u793a\u4e0d\u53d7\u53e6\u4e00\u4e2a\u53e5\u5b50\u7684\u5f71\u54cd<\/font>- \u4e0e\u6211\u4eec\u57fa\u4e8e\u6ce8\u610f\u529b\u7684\u6a21\u578b\u76f8\u53cd\u3002<\/p>\n<p>For TE, Bowman et al. (2015b) use recursive neural networks to encode entailment on SICK (Marelli et al., 2014b). Rocktaschel et al. (2016) present an attention-based LSTM for the Stanford natural language inference corpus (Bowman et al., 2015a). Our system is the first CNN-based work on TE.<br \/> \u5bf9\u4e8eTE\uff0cBowman\u7b49\u4eba\uff082015b\uff09\u4f7f\u7528\u9012\u5f52\u795e\u7ecf\u7f51\u7edc\u7f16\u7801SICK\u4e0a\u7684\u8574\u6db5\uff08Marelli\u7b49\uff0c2014b\uff09\u3002 Rocktaschel\u7b49\u4eba \uff082016\uff09\u63d0\u51fa\u4e86\u65af\u5766\u798f\u81ea\u7136\u8bed\u8a00\u63a8\u7406\u8bed\u6599\u5e93\u7684\u57fa\u4e8e\u6ce8\u610f\u529b\u7684LSTM\uff08Bowman et al\uff0c2015a\uff09\u3002 \u6211\u4eec\u7684\u7cfb\u7edf\u662f\u7b2c\u4e00\u4e2a\u57fa\u4e8eCNN\u7684TE\u5de5\u4f5c\u3002<\/p>\n<p>Some prior work aims to solve a general sentence matching problem. Hu et al. (2014) present two CNN architectures, ARC-I and ARC-II, for sentence matching. ARC-I focuses on sentence representation learning while ARC-II focuses on matching features on phrase level. Both systems were tested on PI, sentence completion (SC) and tweetresponse matching. Yin and Schutze (2015b) propose the MultiGranCNN architecture to model general sentence matching based on phrase matching on multiple levels of granularity and get promising results for PI and SC. Wan et al. (2015) try to match two sentences in AS and SC by multiple sentence representations, each coming from the local representations of two LSTMs. <font color=\"red\">Our work is the first one to investigate attention for the general sentence matching task.<\/font><br \/> \u4e00\u4e9b\u5148\u524d\u7684\u5de5\u4f5c\u65e8\u5728\u89e3\u51b3\u4e00\u822c\u53e5\u5b50\u5339\u914d\u95ee\u9898\u3002 \u80e1\u7b49\u4eba\uff082014\uff09\u63d0\u51fa\u4e24\u79cdCNN\u67b6\u6784\uff0cARC-I\u548cARC-II\uff0c\u7528\u4e8e\u53e5\u5b50\u5339\u914d\u3002 ARC-I\u4fa7\u91cd\u4e8e\u53e5\u5b50\u8868\u793a\u5b66\u4e60\uff0c\u800cARC-II\u4fa7\u91cd\u4e8e\u77ed\u8bed\u7ea7\u522b\u7684\u5339\u914d\u7279\u5f81\u3002 \u4e24\u4e2a\u7cfb\u7edf\u90fd\u5728PI\uff0c\u53e5\u5b50\u5b8c\u6210\uff08SC\uff09\u548ctweetresponse\u5339\u914d\u4e0a\u8fdb\u884c\u4e86\u6d4b\u8bd5\u3002 Yin\u548cSchutze\uff082015b\uff09\u63d0\u51fa\u4e86MultiGranCNN\u67b6\u6784\uff0c\u4ee5\u57fa\u4e8e\u591a\u4e2a\u7c92\u5ea6\u7ea7\u522b\u7684\u77ed\u8bed\u5339\u914d\u5bf9\u4e00\u822c\u53e5\u5b50\u5339\u914d\u8fdb\u884c\u5efa\u6a21\uff0c\u5e76\u4e3aPI\u548cSC\u83b7\u5f97\u6709\u5e0c\u671b\u7684\u7ed3\u679c\u3002 Wan\u7b49\u4eba\uff082015\uff09\u5c1d\u8bd5\u901a\u8fc7\u591a\u4e2a\u53e5\u5b50\u8868\u793a\u6765\u5339\u914dAS\u548cSC\u4e2d\u7684\u4e24\u4e2a\u53e5\u5b50\uff0c\u6bcf\u4e2a\u53e5\u5b50\u6765\u81ea\u4e24\u4e2aLSTM\u7684\u5c40\u90e8\u8868\u793a\u3002 <font color=\"red\">\u6211\u4eec\u7684\u5de5\u4f5c\u662f\u7b2c\u4e00\u4e2a\u8c03\u67e5\u4e00\u822c\u53e5\u5b50\u5339\u914d\u4efb\u52a1\u7684\u6ce8\u610f\u529b\u7684\u5de5\u4f5c\u3002<\/font><\/p>\n<p><strong>Attention-Based DL in Non-NLP Domains.<\/strong><br \/> Even though there is little if any work on attention mechanisms in CNNs for NLP, attention-based CNNs have been used in computer vision for visual question answering (Chen et al., 2015), image classification (Xiao et al., 2015), caption generation (Xu et al., 2015), image segmentation (Hong et al., 2016) and object localization (Cao et al., 2015).<br \/> <strong>\u975eNLP\u57df\u4e2d\u57fa\u4e8e\u6ce8\u610f\u7684DL\u3002<\/strong><br \/> \u5c3d\u7ba1\u5bf9\u4e8eNLP\u7684CNN\u4e2d\u5173\u6ce8\u673a\u5236\u7684\u7814\u7a76\u5f88\u5c11\uff0c\u4f46\u57fa\u4e8e\u6ce8\u610f\u529b\u7684CNN\u5df2\u88ab\u7528\u4e8e\u8ba1\u7b97\u673a\u89c6\u89c9\u4e2d\u7684\u89c6\u89c9\u95ee\u7b54\uff08Chen et al\uff0c2015\uff09\uff0c\u56fe\u50cf\u5206\u7c7b\uff08Xiao et al\uff0c2015\uff09\uff0c \u6807\u9898\u751f\u6210\uff08Xu et al\uff0c2015\uff09\uff0c\u56fe\u50cf\u5206\u5272\uff08Hong et al\uff0c2016\uff09\u548c\u5bf9\u8c61\u5b9a\u4f4d\uff08Cao et al\uff0c2015\uff09\u3002<\/p>\n<p>Mnih et al. (2014) apply attention in recurrent neural networks (RNNs) to extract \u201cinformation from an image or video by adaptively selecting a sequence of regions or locations and only processing the selected regions at high resolution. Gregor et al. (2015) combine a spatial attention mechanism with RNNs for image generation. Ba et al. (2015) investigate attention-based RNNs for recognizing multiple objects in images. Chorowski et al. (2014) and Chorowski et al. (2015) use attention in RNNs for speech recognition.<br \/> Mnih\u7b49\u4eba\uff082014\uff09\u5c06\u6ce8\u610f\u529b\u5e94\u7528\u4e8e\u9012\u5f52\u795e\u7ecf\u7f51\u7edc\uff08RNN\uff09\uff0c\u901a\u8fc7\u81ea\u9002\u5e94\u5730\u9009\u62e9\u4e00\u7cfb\u5217\u533a\u57df\u6216\u4f4d\u7f6e\u6765\u63d0\u53d6\u201c\u6765\u81ea\u56fe\u50cf\u6216\u89c6\u9891\u7684\u4fe1\u606f\uff0c\u5e76\u4e14\u4ec5\u4ee5\u9ad8\u5206\u8fa8\u7387\u5904\u7406\u6240\u9009\u533a\u57df\u3002Gregor\u7b49\uff082015\uff09\u5c06\u7a7a\u95f4\u6ce8\u610f\u673a\u5236\u4e0eRNN\u7ed3\u5408\u7528\u4e8e\u56fe\u50cf\u751f\u6210\u3002 Ba\u7b49\u4eba \uff082015\uff09\u7814\u7a76\u57fa\u4e8e\u6ce8\u610f\u529b\u7684RNN\uff0c\u7528\u4e8e\u8bc6\u522b\u56fe\u50cf\u4e2d\u7684\u591a\u4e2a\u5bf9\u8c61\u3002 Chorowski\u7b49\uff082014\u5e74\uff09\u548cChorowski\u7b49\u4eba\uff082015\uff09\u5728RNN\u4e2d\u4f7f\u7528\u6ce8\u610f\u529b\u8fdb\u884c\u8bed\u97f3\u8bc6\u522b\u3002<\/p>\n<p><strong>Attention-Based DL in NLP.<\/strong><br \/> Attention-based DL systems have been applied to NLP after their success in computer vision and speech recognition. They mainly rely on RNNs and end-to-end encoderdecoders for tasks such as machine translation (Bahdanau et al., 2015; Luong et al., 2015) and text reconstruction (Li et al., 2015; Rush et al., 2015). Our work takes the lead in exploring attention mechanisms in CNNs for NLP tasks.<br \/> ** NLP\u4e2d\u57fa\u4e8e\u6ce8\u610f\u529b\u7684DL\u3002**<br \/> \u5728\u8ba1\u7b97\u673a\u89c6\u89c9\u548c\u8bed\u97f3\u8bc6\u522b\u65b9\u9762\u53d6\u5f97\u6210\u529f\u540e\uff0c\u57fa\u4e8e\u6ce8\u610f\u529b\u7684DL\u7cfb\u7edf\u5df2\u5e94\u7528\u4e8eNLP\u3002 \u4ed6\u4eec\u4e3b\u8981\u4f9d\u9760RNNs\u548c\u7ec8\u7aef\u5230\u7ec8\u7aef\u7684encoderdecoders\u4efb\u52a1\uff0c\u5982\u673a\u5668\u7ffb\u8bd1\uff08Bahdanau\u7b49\u4eba\uff0c2015\u5e74;\u9648\u5fb7\u826f\u7b49\u4eba\uff0c2015\u5e74\uff09\u548c\u6587\u672c\u91cd\u5efa\uff08\u674e\u7b49\u4eba\uff0c2015\u5e74;Rush\u7b49\u4eba\uff0c2015\u5e74\uff09\u3002\u6211\u4eec\u7684\u5de5\u4f5c\u7387\u5148\u63a2\u7d22CNN\u4e2d\u9488\u5bf9NLP\u4efb\u52a1\u7684\u6ce8\u610f\u673a\u5236\u3002<\/p>\n<h2>3 BCNN: Basic Bi-CNN<\/h2>\n<p>We now introduce our basic (non-attention) CNN that is based on the <strong>Siamese<\/strong> architecture (Bromley et al., 1993), i.e., it consists of two weightsharing CNNs, each processing one of the two sentences, and a final layer that solves the sentence pair task. See Figure 2. We refer to this architecture as the BCNN. The next section will then introduce the ABCNN, an attention architecture that extends the BCNN. Table 1 gives our notational conventions.<br \/> \u6211\u4eec\u73b0\u5728\u4ecb\u7ecd\u57fa\u4e8e<strong>Siamese<\/strong>\u67b6\u6784\u7684\u57fa\u672c\uff08\u975e\u6ce8\u610f\uff09CNN\uff08Bromley\u7b49\uff0c1993\uff09\uff0c\u5373\u5b83\u7531\u4e24\u4e2a\u6743\u91cd\u5171\u4eabCNN\u7ec4\u6210\uff0c\u6bcf\u4e2a\u5904\u7406\u4e24\u4e2a\u53e5\u5b50\u4e2d\u7684\u4e00\u4e2a\uff0c\u6700\u540e\u4e00\u5c42\u89e3\u51b3\u53e5\u5b50\u5bf9\u4efb\u52a1\u3002 \u53c2\u89c1\u56fe2.\u6211\u4eec\u5c06\u6b64\u4f53\u7cfb\u7ed3\u6784\u79f0\u4e3aBCNN\u3002 \u63a5\u4e0b\u6765\u7684\u90e8\u5206\u5c06\u4ecb\u7ecdABCNN\uff0c\u8fd9\u662f\u4e00\u79cd\u6269\u5c55BCNN\u7684\u6ce8\u610f\u529b\u67b6\u6784\u3002 \u88681\u7ed9\u51fa\u4e86\u6211\u4eec\u7684\u7b26\u53f7\u7ea6\u5b9a\u3002<\/p>\n<p>In our implementation and also in the mathematical formalization of the model given below, we pad the two sentences to have the same length <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s = m a x ( s 0 , s 1 ) s = max(s_0, s_1) <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.43056em; vertical-align: 0em;\"><\/span><span class=\"mord mathdefault\">s<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">=<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mord mathdefault\">m<\/span><span class=\"mord mathdefault\">a<\/span><span class=\"mord mathdefault\">x<\/span><span class=\"mopen\">(<\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span>. However, in the figures we show different lengths because this gives a better intuition of how the model works.<br \/> \u5728\u6211\u4eec\u7684\u5b9e\u73b0\u4e2d\u4ee5\u53ca\u4e0b\u9762\u7ed9\u51fa\u7684\u6a21\u578b\u7684\u6570\u5b66\u5f62\u5f0f\u5316\u4e2d\uff0c\u6211\u4eec\u5c06\u4e24\u4e2a\u53e5\u5b50<font color=\"blue\">\u586b\u5145\u4e3a\u5177\u6709\u76f8\u540c\u7684\u957f\u5ea6<\/font><span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> s = m a x ( s 0 , s 1 ) s = max(s_0, s_1) <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.43056em; vertical-align: 0em;\"><\/span><span class=\"mord mathdefault\">s<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">=<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 1em; vertical-align: -0.25em;\"><\/span><span class=\"mord mathdefault\">m<\/span><span class=\"mord mathdefault\">a<\/span><span class=\"mord mathdefault\">x<\/span><span class=\"mopen\">(<\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mpunct\">,<\/span><span class=\"mspace\" style=\"margin-right: 0.166667em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">s<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">1<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mclose\">)<\/span><\/span><\/span><\/span><\/span>\u3002 \u4f46\u662f\uff0c\u5728\u56fe\u4e2d\u6211\u4eec\u663e\u793a\u4e86\u4e0d\u540c\u7684\u957f\u5ea6\uff0c\u56e0\u4e3a\u8fd9\u6837\u53ef\u4ee5\u66f4\u597d\u5730\u76f4\u89c2\u5730\u4e86\u89e3\u6a21\u578b\u7684\u5de5\u4f5c\u539f\u7406\u3002<\/p>\n<p>We now describe the BCNN\u2019s four types of layers: input, convolution, average pooling and output.<br \/> \u6211\u4eec\u73b0\u5728\u63cf\u8ff0BCNN\u7684\u56db\u79cd\u7c7b\u578b\u7684\u5c42\uff1a<strong>\u8f93\u5165\uff0c\u5377\u79ef\uff0c\u5e73\u5747\u6c60\u548c\u8f93\u51fa\u3002<\/strong><br \/> <img decoding=\"async\" src=\"https:\/\/img.mushiming.top\/app\/mushiming_com\/81a306415b4d43399735ba1355f4f042.png\" =\"\" =\"\u5728\u8fd9\u91cc\u63d2\u5165\u56fe\u7247\u63cf\u8ff0\" alt=\"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs (https:\/\/mushiming.com\/)  \u7b2c2\u5f20\" title=\"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs  \u7b2c2\u5f20-\u7a46\u4e16\u660e\u535a\u5ba2\" ><br \/> <strong>Input layer.<\/strong><br \/> In the example in the figure, the two input sentences have 5 and 7 words, respectively. Each word is represented as a <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> d 0 d_0 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.84444em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span><\/span>-dimensional precomputed word2vec (Mikolov et al., 2013) embedding, <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> d 0 = 300 d_0 = 300 <\/span><span class=\"katex-html\"><span class=\"base\"><span class=\"strut\" style=\"height: 0.84444em; vertical-align: -0.15em;\"><\/span><span class=\"mord\"><span class=\"mord mathdefault\">d<\/span><span class=\"msupsub\"><span class=\"vlist-t vlist-t2\"><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.301108em;\"><span class=\"\" style=\"top: -2.55em; margin-left: 0em; margin-right: 0.05em;\"><span class=\"pstrut\" style=\"height: 2.7em;\"><\/span><span class=\"sizing reset-size6 size3 mtight\"><span class=\"mord mtight\">0<\/span><\/span><\/span><\/span><span class=\"vlist-s\">\u200b<\/span><\/span><span class=\"vlist-r\"><span class=\"vlist\" style=\"height: 0.15em;\"><span class=\"\"><\/span><\/span><\/span><\/span><\/span><\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><span class=\"mrel\">=<\/span><span class=\"mspace\" style=\"margin-right: 0.277778em;\"><\/span><\/span><span class=\"base\"><span class=\"strut\" style=\"height: 0.64444em; vertical-align: 0em;\"><\/span><span class=\"mord\">3<\/span><span class=\"mord\">0<\/span><span class=\"mord\">0<\/span><\/span><\/span><\/span><\/span>. As a result, each sentence is represented as a feature map of dimension <span class=\"katex--inline\"><span class=\"katex\"><span class=\"katex-mathml\"> d 0 \u00d7 <\/span><\/span><\/span><\/p>\n","protected":false},"excerpt":{"rendered":"ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs\u6587\u7ae0\u76ee\u5f55AbstractAbstractHowtomodelapai...","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"_links":{"self":[{"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/posts\/7861"}],"collection":[{"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/comments?post=7861"}],"version-history":[{"count":0,"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/posts\/7861\/revisions"}],"wp:attachment":[{"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/media?parent=7861"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/categories?post=7861"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/mushiming.com\/wp-json\/wp\/v2\/tags?post=7861"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}