algorithms for nlp
play

Algorithms for NLP CS 11-711 Fall 2020 Lecture 9: CRFs, neural - PowerPoint PPT Presentation

Jul 23, 2023 •327 likes •2.04k views

Algorithms for NLP CS 11-711 Fall 2020 Lecture 9: CRFs, neural sequence labeling Emma Strubell Announcements Project 2 released today after class : sequence labeling. Due: October 16. You will implement part-of-speech taggers for

  1. <latexit sha1_base64="LEiuqxhwrDWuN4pQW8a63QJUygI=">AEvXiclVLdbtMwFE5GgVH+NrjkxlBNa2GbmoENxMTcMHNRJHYD5pL5TgnrTX/RLaztbLyIDwNt/AIvA1O0k3LOpCwFPvk+DvnfMfnizPOjO3f4dLN1o3b91evtO+e+/+g4crq48OjMo1hX2quNJHMTHAmYR9yHo0wDETGHw/jkfXl/eAraMCW/2FkGQ0HGkqWMEutdo9Vwu72GB4Z1cSzcWbGBynNW9NAOwiYXmDPBrBk5sRMV39zei6goAdhOwJICYZoW4WkxUWG2UiUmxObkf8TvfbaoFtnxYIlqIZVBVJNqMwzbrXUegVrkGhcq/7okwiLIidUMLd1/O6veIvidbLTO3/7DG7hF7oZ7TS6W/1q4UWjWhudIL5GoxWlzROFM0FSEs5MeY46md26Ii2jHLw7HIDGaEnZAzH3pREgBm6aroFWvOeBKVK+09aVHkvRzgijJmJ2CPLVzFX70rndXfHuU3fDB2TW5B0rpQmnNkFSqlghKmgVo+8wahmnmuiE6In5n1gvKyuVRmAvwUbLMRKobOpFX1BqVYFM3gad1oG2uQcEaVEQmzx1OiWB8lkBKcm5LMaTn9nXvtZGcszMn+4iJQeLlWZjJgnkFpcbk23PyYWV3sbfwA/IA17nvWnDSxSnsmRI8FmRZ+YGP8FJfmv5BMXiC92WzLVQR8M+W7qAykK2otc2UAx2Ot8qxBeCG+IuoTkNSPocZDM6xGeJVGVzW5aBxsb0Uvt7Y/v+rsvpvrdTl4EjwLukEUvA52g4/BINgPaPg9/BH+DH+13ragxVuyhi6F85jHQWO1zv4AU5+hA=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! exp( Ψ ( w , y )) M +1 P ( y | w ) = X Ψ ( w , y ) = ψ ( w , y m , y m − 1 , m ) P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) m =1 y’ is an entire sequence decompose into local scores M +1 X y ) = θ · f ( w , y m , y m − 1 , m ) m =1 4

  2. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 5

  3. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  4. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 VB? Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  5. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 VB? Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  6. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 f ( w , VB, MD, 3) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  7. <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  8. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  9. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # labels m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  10. <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # features # labels m =1 K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  11. <latexit sha1_base64="PlZBZr6ihz0X20hclBQ9g8VkZw=">AEzHiclVJb9MwFE5GgRFuGzyYpimtWyrmoEL5PGRQhpGhSJXdBcKsc5a3ZTmQ72yor/wWfg2vwL/BSbpWQcSlup8Pf7Od+5RBln2vR6v/25a63rN27O3wpu37l7/7C4oM9neaKwi5NeaoOIqKBMwm7hkOB5kCIiIO+9HRm/J9/xiUZqn8bCYZDAQZSZYwSowzDRf9V8Ey7mvWxpGwJ8UaKr+ToM2Eda5wJwJZvTQis2w+Gp3VsOiJGAzBkMKhGmcmsolKc4jTIaivKxYD90/0QmW+06KhYsRjWtSpAoQi2G06x9lYROYRsSKvOKS8okwoKYMSXcfjnL2yn+EmiljPTfRWYX2DMFtbfXw057G62idx0UDBeWet1edAsCKdgyZue/nBxTuE4pbkAaSgnWh+GvcwMLFGUQ5FgHMNGaFHZASHDkoiQA9sNewCLTtLjJUuZ80qLJe9LBEaD0RkWOWTdKX30rjVW+HuUleDiyTW5A0jpRknNkUlRuDoqZAmr4xAFCFXNaER0TN0Lj9s1+EKaMfBjM1CqBhYnVTZG5IiUTSdT+tCA6xAwglNhSAyfmpxQgTjkxgSknNT7kZyhq/q1p8zDI9bd15SA4Gp4qNmCScQ2JweTXN7jM2uLoD/BbcgBTsONUfM1DEpMopIWokyGnhBjbCj3EJ/8Vk8pzpYLMsWwlwxZR9STOQtqhXm6cacDRSaZ41BM/4V0JdAJK4MdR8aLrVDLel4eWdnAV7G93wWXfj0/OlrdfTfZ3HnlPvLYXei+8Le+91/d2Pep/93/4P/1frQ8t07KtoqbO+VOfh17jtL79ATypCw=</latexit> <latexit sha1_base64="6QgzZybjFg9uG/YPephMOPdGJY8=">AHcnicnVTdbts2FZTr+68nzbdXvDLghirV5gdQXWmwJFt4vdNPMAp+kQOgJFUTYdUhJIyolBcO+5J9gL7AFGUrJrOW4CTICp43PO953vHFJMSkalGg7/vrd3v/PFg+7DL3tf3Nt48e7z/5ItKYHKC1aIjwmShNGcnCqGPlYCoJ4wshZcvmLi58tiJC0yMdqWZIJR9OcZhQjZV3x/v1/eodwhpRWJo4ucvAGQCSmHF3HWjmHAaO+NwDkNAVXzgx7hzfTYCYQ1qO+z/C5PhA28NCsYqHZBf8sHfYxMBG0OLUhTXANo1jTV9E5mL3CXSJot64O3ZNmtFqumPkQn/xyw+cW8P5PKwM7Si1j156Gr6mk8GRh+Wy4T0NkXOTxHpu+kWsnFbEyhlao2TFV6gmcgvUtTk2Gy2Oa710Q9qVkzY3F5q+mJtBrfTS/XV6w2a+61w6WPfiaP1xgOS67EM1IwoBiNCgcxBndQG+2agYxyqTt4ghSezKQmHE9J/G7KQ9GtxB6vSOa7Vn2xJHkvbPBmC8Xp8BGxtsFkctBFHjvgTvr0R+sTA0obu7hn0PAdMuL6yQfdemhXbWg73pO9fRMYl1BMwzQicJzNrhmXM3aK5ZR8AHsaPD4bHQ/+Am0bUGAdB84zi/T0B0wJXnOQKMyTleTQs1UQjoShmxPRgJUmJ8CWaknNr5ogTOdH+SjLg0HpSkBXC/nIFvHcToRGXcskTm+lmK7djzrkrdl6p7PVE07ysFMlxXSirGFAFcPcbSKkgWLGlNRAW1GoFeIbsTit7C9qd2igzI2xBVLsRzCdaZr56S1LCTRt8XTfag4Lk5AoXnKM8/UHDHKlinJUMWUO03Zyt41r0G6oKVsRremZETBQtApzRFjJFPQLW23fc0U9GsP/krsBgny3qr+vSQCqUJYJfWHbeyGTeFzd6eY2zJpvs60Zrst7QXYZtxcipLk2tRfBCskgclUFXZEnwD74VaApTZbajzSRtWZ9hTGm2fyZvGh5fH0U/HL/94dfD2XNeHwbPgu+DfhAFPwdvg9+CUXAa4M5JR3VM568H/3afdp93m8O9d6/BfBe0nu7gPxLUizA=</latexit> <latexit sha1_base64="wMAXQnpawC1ge/9hOKMhUb3NFJc=">AExXiclVJb9MwFE5GgRFuGzyYpimtWxUzUACIU2aAESmigSu6C5VI5z0lqznch2ulVWxG/h1/AKj/wbnKSblnUgYanO1+PvnPOdS5Rxpk2v9tfuNK6eu364o3g5q3bd+4uLd/b02muKOzSlKfqICIaOJOwa5jhcJApICLisB8dvS7f9yegNEvlZzPNYCDISLKEUWKcabjsvwxWcV+zNo6EPS42UPmdFh20hbDOBeZMKOHVmyFxVe7sx4WJQGbMRhSIEzj1FQuSXEWYToU5WXFk9D9E51gtd+uo2LBYlTqgSJItRiOMnal0noFLYhoTKvuaRMIiyIGVPC7ZfTvJ3iL4HWykj/XWR2j1X0Ae0jt6iYLi0uv2qoPmQTgDK97s9IfLCwrHKc0FSEM50fow7GVmYIkyjHIoApxryAg9IiM4dFASAXpgqykXaNVZYpSkyv2kQZX1vIclQupiByz7I6+FYaL3s7zE3yYmCZzHIDktaJkpwjk6JyZVDMFDpw4QqpjTiuiYuNkZt1ius+fSjIFPwDQLoWJgdVJlb0iKRNF0PqkLDbACc0FYLI+LHFCRGMT2NISM5NuRTJKb6sXxvxhGV61rqzkBwMThUbMUk4h8Tg8mqa3WdscHUH+A24ASnYcao/ZqCISZVTQtRIkJPCDWyEH+IS/ovJ5BnTwWZthLgin7kmYgbVHvNE814Gik0jxrCJ7zr4S6ACRxY6j50HSrGW5Lw4s7OQ/2Nrvh0+7mp2cr269m+7roPfAeW0v9J572957r+/tetT/7v/wf/q/Wu9aomVak5q64M987nuN0/r2BwZQops=</latexit> Conditional random fields (CRFs) ■ Linear-chain CRFs : Globally-normalized discriminative sequence labeling models! M +1 X Ψ ( w , y ) = θ · f ( w , y m , y m − 1 , m ) # features # labels m =1 ( K − 1)( K + F ) K + F f ( w , VB, MD, 3) = f ( w 1 , w 2 , w 3 , w 4 , VB, MD) 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD y m-1 MD VB? y NNP Janet will back the bill <s> w w m-2 w m-1 w m w m+1 5

  12. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) 6

  13. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) 6

  14. <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y 6

  15. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) y = argmax ˆ log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) 6

  16. <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) y = argmax ˆ log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) 6

  17. <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y 6

  18. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y M +1 X = argmax ψ ( w , y m , y m − 1 , m ) y m =1 6

  19. <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> <latexit sha1_base64="Hl4SzGDfq/v1/mRkJxB+iyWEgs4=">AH9nicnVTPb9s2FayZG69X0137IVbEFha3cDqCmyXAkW3wy7tPMBpOoSOQFGUTYeUBIpybBD8V3obdt3fstv+kx1HUrJrOU4CTICl5/e+7+P3HgnGBaOlHAz+2dv/5ODw086Dh93Pv/iy68eHT1+V+aVwOQM5ywX72NUEkYzciapZOR9IQjiMSPn8dVPtn4+J6KkeTaSy4KMOZpkNKUYSZOKjg6uidwiqSOgovM/ASQCQmHC0iJW1Cg6HvAgA5TcC1DYPuyU0YTAXCaug7hMO6QtDQA72qBXoX/XZi96SpgY2i4ReFyBcAmncSKfoy1JeZBdIGR3xbrRBrUQVfRbq4H/M4qP29kBudwZ2Lj2PpD8L7KpOxhHfzo2eKfs0QCY103GkZtrPI2m9IlZM0ZpVnzFaip3UG2bI73R4qj2SzesXVtrM32p6NOZ7tdOr+xf6zdo5rvG0v6FyvrjgMki8KHckokAhAnuQSpfTvkdbKNgMZ5VSWposepOZkIDnFiKnftd4p2+vfI2r9jmq359sWhyX1z/tgtL30qAfM2mBzcdBm9KzwR357I9RbDQtTur9n0IjAmKtrU7XfpV7Jrf1wp/rmagtoB6BbmZgM6leKywjbl+KG/k+4EF36NeibgI1atcYthxsT8Sle3p7LA0t0LcI9axS9Oh4cDpwD7gZhE1w7DXPMDraFzDJcVJjFDZXkRDgo5VkhIihnRXViVpED4Ck3IhQkzxEk5Vu5a1ODEZBKQ5sL8MglcdpOhEC/LJY8N0jZSbtdsclftopLpj2NFs6KSJMP1QmnFgMyBvWNBQgXBki1NgLCgxivAU2TGLM1NbDZ6Y5kpYXMi241gPlZl6lZvWYq5bpMXdaNdKEhGrnHOcqS7xRMEadsmZAUVUza/UtX8a59ZM5LcpmdGtJRiTMBZ3QDFGUgntq502n6mE7t2FPxOzQYK8Ma5/LYhAMhfGSX25aLNhE/iNvdf0XUiarZEmbLelnAHTjJ1LXpBM6fr4sbwkMJ6IvCpahm/wnVEjgFKzDTWetGk1wpzScPtM3gzePT8Nvz9/tuL41evm/P6wHvifev5Xuj94L3yfvG3pmHD/4+Pdw73C/s+h86PzR+bOG7u81nK+91tP56z9G0Lzs</latexit> <latexit sha1_base64="DypLI6QWGLOj7fmyWz0DYpUsnmg=">AFfXiclVNdb9MwFM3KCqN8bfDIi2Gq1rKuagbSeJk0AUJI0SR2JfmUjmO01qz48h2tlWxSv8RP4GfwCcpO2atSDNUpyb63Pf6JkgYVbrT+bVUubNcvXtv5X7twcNHj5+srj09UiKVmBxiwYQ8CZAijMbkUFPNyEkiCeIBI8fB+fvs/PiCSEVF/FWPEtLjaBDTiGKknau/tvS7VodRsw4ObStkD2Htkm2AVQpRwyqlWfcN3fvNHGz6NgNAPSQaWQBxKHQeEtkpw6jPs83wLd98Wat3m0UrJDTEBSwPEkETaQXCWNRKa1pQk5O4Nl5TGAHKkhxgxczrJ27T/INrImG5dZDKDni+osb/lNxv7YBN8bALHPUS6kGdtxonkgKOr/tQFmRiASRPAbBdqi9ATpaOx0qxbWznJrftxzbVxTdZfXe+0O/kC84Y/Nta98er21yoShgKnMQaM6TUmd9JdM8gqSlmxNZgqkiC8DkakDNnxogT1TP5dFpQd54QREK6J9Yg985GMSVGvHAIbMq1M2zLno7CzV0dueoXGSahLjIlGUMqAFyEYdhFQSrNnIGQhL6rQCPERu5rT7IdytzaQZEnZBdLkQzHtGRXn2kqSA23LwVFoDUoSk0sOEdx+MrACHKRiGJUMp0NszRxF7Ur1Z4QRM1bt2UkhENhaQDGiPGSKRhtpXd7jXUMN9r8ANxFyTJgVP9OSESaSGdkmLErLuwAXwBM/N/SBpPkc4sl2VyAa6YrC8iIbGxewxoQgMBlKkSUnwXHwu1BGgyF1DgSflsALhptS/OZPzxtF23/d3v7yZn3v3XheV7zn3kuv4fnejrfnfK63qGHK6eV75UflZ/Lf6r1aqvaLqCVpXHM6+0qjt/AWSk5PE=</latexit> Conditional random fields (CRFs) exp( Ψ ( w , y )) ■ Decoding: Direct application of Viterbi! P ( y | w ) = P exp( Ψ ( w , y 0 )) y 0 ∈ Y ( w ) ˆ y = argmax log P ( y | w ) y X exp Ψ ( y 0 , w ) = argmax Ψ ( y , w ) − log y y 0 ∈ Y ( w ) same for all settings of y = argmax Ψ ( y , w ) y M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) y m =1 6

  20. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 7

  21. <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 M +1 X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 7

  22. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 7

  23. <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) y 1: M m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 m =1 7

  24. <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 m =1 7

  25. <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  26. <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 same subproblem! score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  27. <latexit sha1_base64="cFZeFhMRSODTiOyXYHFD+C/9s=">AGOHiclVPdbtMwFM5WCiP8bcAdN4apWst+1Awk0KRJE5sQ0jRJPaD5hI5jtNas5PIdrZVlh+JF+BNuOMOcsT4CRt16wFaZbqnB5/5zufj8JUkalard/zM3XbtVv31m46967/+Dho8Wlx0cyQmhzhiTgJkCSMxuRQUcXISoI4gEjx8HZbn5+fE6EpEn8WQ1S0uWoF9OIYqSsy1+a/+Y2YEfSJgy4vjBrIP8OTAtsAygzDhnlVElf823PfNUHq57JAVD1iUIGQBwmqgiJzJh4PN803zds/94y210miUr5DQEJaxIEAmENSXaXOWhJbRFQmFe8UmpTGAHKk+Rkx/GeVtmX8QreRMN75kOoGevlBzf91rNfBKnjfApa7j1Qpz5icE4keR5f+2AVZ0gOjIoDJKriNWfCR1MFQal6u9YLlxgW54lq5IpuVNWe+QW2pc+bk+6WCyfofO1tHVjS6aKXJ0XpZ+CviZhK4i8utzfaxQLThjc0lp3h6tgGFzBMcMZJrDBDUp567VR1NRKYkaMCzNJUoTPUI+cWjNGnMiuLgbLgIb1hCBKhP3FChTeyQiNuJQDHlhkXn95/Sx3zjo7zVT0tqtpnGaKxLhMFGUMqATkUwpCKghWbGANhAW1WgHuIzsuys6ybiJNH3CzomqXgTzrpZRkb0iKeCmGnxZXtSFgsTkAiecozh8qWGEOGWDkEQoYyqfw2hkz6rXWnhOUzks3ZiSEQUTQXs0RoyRSMF8q7rtp69gsbtwj9gHEuTAqv6YEoFUIqySsk2NfbAefJ63jPkfksZjpDWr19KFAHuZvC5JSmJtyqlhiSQw6IkSyuCp+ILoZYARfYZSjyphpUI26Xe9Z6cNo42N7xXG5ufXi/vBv264LzHnhNB3PePsOB+cjnPo4NrT2lZt7ZX/17/Wf9V/1C5+eGMU+cyqr/+Qtf4ih5</latexit> <latexit sha1_base64="DlaVeuBnBAV5S8rb4kmojlkndes=">AFvniclVPdbtMwFM4KhRH+NrjkxjBVa9mPmg0JbiZNgBDSNFEk9oPmUjmO01qz48h2tlZWXoSn4RbegLfBTtquWQtiluqcHn/nO985Pg5TRpVut38v1W7drt+5u3zPv/g4aPHK6tPjpXIJCZHWDAhT0OkCKMJOdJUM3KaSoJ4yMhJeP7OnZ9cEKmoSL7oUq6HPUTGlOMtHX1Vmu7fgN2FG3CkJvLfBO47yhvgT0AVcYho5xq1TN8L8i/mcONIHcAqAdEoxAHAldhMT5lGHU424zfCuw/3jLb3SaJSvkNAIlrEgQS4QNJMO0uUhCKzcVCYV73SalCYAc6QFGzHyd5G3lfyFad0w3LjKdQc8X1DzYClrNA7ABPrSA5R4gXcrLc8eJZJ+jYW/qgkz0waQJYLYLfmMRfCJ1NJbq2rVsNy4IVdc61dk/nxSR/zfDeitrLW328UC80YwNta8erYMZMwEjJNGYIaXOgnaquwZJTEjuQ8zRVKEz1GfnFkzQZyorinGOwcN64lALKT9JRoU3tkIg7hSIx5apOuCun7mnIvOzjIdv+kamqSZJgkuE8UZA1oA91ZARCXBmo2sgbCkVivA2SHVtsXZa9Js2AsAuiq4Vg3jUqLrJXJIU8rwYPy0J9KElCLrHgHCXRSwNjxCkbRSRGdPuNcQTe1G/NqMLmqpx6aUjGgoJO3TBDFGYg3dVnXbz0DYvfhe2IvSJDq/pTSiTSQlol5bTk9sL68Dl05r+QNJkirVktyxQCbDGuLyIlicnL2WVCERj2pcjSiuC5+EKoJUCxvYST6phJcJOaXB9JueN453tYHd75/Ortf2343ld9p5L7ymF3ivX3vo9fxjxc+17UftZ+1Xfr8d1XhcltLY0jnqVZ9+AfhCPx+</latexit> <latexit sha1_base64="WGAjEtshRFBM14UTBXAPpJKzjY=">AGpHiclVRdb9MwFM1WKN8bfCEeDFMZS37oBlIKRJEyCENCaKoNvQ3EWO46TW7CSynW1VFIlfwyv8Hf4NdtJ2TVOQFqnO7fW5xf23FjRqXqdP4sLNauXa/fWLrZuHX7zt17yv3D2SUCEx6OGKROHKRJIyGpKeoYuQoFgRxl5FD9/SdmT8I0LSKPymhjHpcxSE1KcYKZ1yVmoPG03YlbQFXZ6eZxvAvIdZG+wAKBMOGeVUSflO3Z2ku6v25kBQDUgCmUAYi9SeYmfTRiGDjdDyjdt/Y+3G81uq2CFnHqgOUCvkA4heQibs2z0M7SkoU8vaZFaQgR2qAEUu/j3Xb2T+I1gzTlRcZT6GrC2rtbdrt1h5YBx/aQHMPkCrsZnhRCLg6MKZpCLAjBuApjuQqM5Dz62OhxZNe3azFmu3JBLrVLsnmqhvkKHdiRDm9Np3UOTvE5qf1mX7NWu17M5L2fg59xUVWZLdNFuhUG24IMhQEjzyUeZBXGan1qpTmKotVpZzl1c5WJ39ANbBHwao1erOyqKAXoQTkKFGZLy2O7Eqp8ioShmJGvARJIY4VMUkGMdhogT2U/zS5yBps54wI+E/oUK5NnpihRxKYfc1Uiz13J2ziTnzR0nyn/dT2kYJ4qEuBDyEwZUBMwXAXhUEKzYUAcIC6q9AjxA+moq/d3QuzolMyDsjKjyQjDvp9LP1UuWXJ6Viy+KhTagICE5xHnKPSepdBHnLKhR3yUMGXuvD+O5/VrwzujsRy1bkLJiIKRoAENEWPEV9AM5bR+DRTMxwZ8T/QGCbKvX+OiUAqEtpJcSUyvWEBfGyOTPY/JA0nSB2Wl5XmBvRiTF+imIRpVtxQFkC3UBESVwyXKnPjWoC5OtKPCkXFYg9Cm1Z89kNTjY3rJfbG1/ebm6+3Z0XpesR9YTq2XZ1itr1/poda2ehWs/aj9rv2q/60/rn+pf670CurgwqnlglZ76yV/XA02e</latexit> <latexit sha1_base64="tmRzwOs7SiKi8INixRl4Jca9kY=">AFB3iclVNb9MwFE5Lga1cdnvkxTBNa9lFzUCl0kTIQ0TRSJXdBcKsc5a3ZcWQ726rgH8Cv4Q3xys/gmT+Ck3RVsw4kLMU+Of7Od+5OEg406bT+VWr32rcvnN3br57/6DhwuLS8tHWqaKwiGVXKqTgGjgLIZDwyHk0QBEQGH4+DsdX5/fA5KMxl/NKMEeoIMYhYxSoxT9ZdqX5pruKtZCwciu7CbKD9Hto12EdapwJwJZnQ/E7u+/ZwdbPg2B2AzBEMswjSUpjCJ7MTDqC/yLRNbvsT7eZat1V6xYKFqIQVASJFaIbhMmndRKFtswqFQr3ugrIYUHMkBKefbqK27Z/cbSe/rvJMp9GxCrf0tv93aRxvobRs18ZCYkp21uUuiBoJc9icqzOUAXdUATRehv7ja2e4UC80K/lhY9car21+qKxKmgqIDeVE61O/k5heRpRhlINt4lRDQugZGcCpE2MiQPeyYk4sWnOaEVSuS82qNBOW2REaD0SgUPm9dX73LlTXenqYle9jIWJ6mBmJaBopQjI1E+dChkCqjhIycQqpjiuiQuO4bN5quN1NhsDPwVQToaKX6aiIXqEUCFs1viwTbWIFMVxQKQSJw6cZjohgfBRCRFJu8rGKruSb6rUZnrNEj0s3cnBYKnYgMWEc4gMzreq2h1Dg4u9id+Aa5CA8f6fQKGKkck3I2rGvYAD/GufgvJIsnSCdW08oKAi6ZvC4ygTiz5avgUgMOBkqmSYXwjH1B1DkgkWtDiYeqWYlwU+pfn8lZ4Whn23+2vfPh+ereq/G8znmPvCdey/O9F96e987reocerf2uz9eX6yuNr41vje+NHyW0XhvbrHiV1fj5BxU+uaY=</latexit> <latexit sha1_base64="HUJof5rs93v8xuaMgJAQitvYE=">AGNXiclVPbhMxEN02BEq4tfDIi6GKmtCLsgWpCKlSuQghVRVBohdUp5HX602s2uV7W0bWf4hfoBv4YE3xCu/gHc3SbNQNRSvJPxzJnj45kgYVTpVuv73HzlRvXmrYXbtTt3791/sLj08ECJVGKyjwUT8ihAijAak31NSNHiSIB4wcBqdvs/PDMyIVFfFnPUhIh6NeTCOKkXau7tL81odthVtwICbc7sGsu/ANsE2gCrlkFOteoavu3bE7O36tsAOo+0cgCiEOh85TIjhEGXZ5thq/7h9v1urtRoEKOQ1BEZYXiCTCBpKLpDGLQtOaEoXcveK0hAjnQfI2a+jOo27V+AVjKka18ymYievlBjd91vNnbBKnjfBA67j3RBz9oME8keRxfdsQsy0QMjEcCkCrX6rPAR1cGQaibXeo5ybUEusVYuwWZVzZD/X4Ft1eWNSa/DhBNwXeO/2nOg06IXJ7n0Lv5aVbuLy62NVr7AtOEPjWVvuNquyUMBU45iTVmSKljv5XojkFSU8yIrcFUkQThU9Qjx86MESeqY/KpsqDuPCGIhHS/WIPcO5lhEFdqwAMXmYmvrp5lzlnx6mOXnYMjZNUkxgXhaKUAS1ANqIgpJgzQbOQFhSxXgPnKzot0gO50nyvQJOyO6fBHMO0ZFefUSpYDbcvJFcdEalCQm51hwjuLwmYER4pQNQhKhlOlsCKORPUuvtfCMJmo3RiSEQ2FpD0aI8ZIpG2ld3u09cw32vwHXEPJMmeY/0xIRJpIR2Tokete7AefJL1i/1XJI3Hkc4sX8vkBNxlMl1EQmJji5FhQhEY9KRIkxLhqfycqANAkXuGIp6U04oI16X+1Z6cNg42N/znG5ufXizvBn264L32HvqNTzf2/J2vA9e29v3cGWpslXZqbyufqv+qP6s/ipC5+eGOY+80qr+/gO+OCZh</latexit> Conditional random fields (CRFs) ■ Decoding: Direct application of Viterbi! ψ includes transition and emission/unary scores M +1 X = argmax ψ ( w , y m , y m − 1 , m ) = s m ( y m , y m − 1 ) ˆ y y m =1 same subproblem! score of best tag sequence M +1 of length M X max y 1: M Ψ ( w , y 1: M ) = max s m ( y m , y m − 1 ) score of best tag sequence y 1: M of length M-1 m =1 M X = max y M s M +1 ( h / s i , y M ) + max s m ( y m , y m − 1 ) y 1: M − 1 score of m =1 most probable extension y M 7

  28. Learning in CRFs 8

  29. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 8

  30. <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) 8

  31. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 sum over all possible labelings N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) 8

  32. <latexit sha1_base64="qyQNuw9LW72v28UgaIfcr0GnkE=">AG6HiclVTdbtMwFM5WKP8bXDJjWGqlrB1NAMJBJo0AUJI06BIbAPNXeW6TmpmJ5HtbKusvAN3iFuehZfgFbiFB8BOuq5pC9Is1Tk9/s53Ph8fu5swKlWz+XNuvnLpcvXKwtXates3bt5aXLq9J+NUYLKLYxaLj10kCaMR2VUMfIxEQTxLiP73aOXdn3/mAhJ4+iDGiSkzVEY0YBipIyrs1Rp1+qwJakLu1yfZGvAfgeZBzYBlCmHjHKqZEfzT871DurfmYBUPWJQhmAuBerPCTIRgyDreT5g3f/ONerd5yC1bIaQ8UsDxBIBDWkJwm7iwJXqZLEnL3iklKIwA5Un2MmP50ltfL/kG0YpkuvMlkD29IXe74XvuNlgFrz1guPtIFfKyzHIiEXJ02hm5ItDcFYEMF6FWn0W/EzqYCjVlquRs1y4IOdcK+dks7Ja5gtUYFN2uDvuNj4xtfR/rMdwzpd9WIlr/0M/ISKGVkm40yUqYUFu5ChKGTkoYQiN/Iwm2t1OpchK2ebkQsSxozORoGjFve2dJaH2qVe6UQLz3NQ3BCvs7jcXG/mA0wb/tBYdoaj1VmaF7AX45STSGpDzwm4lqayQUxYxkNZhKkiB8hEJyYMwIcSLbOn8IMlA3nh4IYmF+kQK5dzxCIy7lgHcN0vaLnFyzlrB6kKnrY1jZJUkQgXiYKUARUD+6qAHhUEKzYwBsKCGq0A95G53sq8PaYzxtL0CTsmqrwRzNtaBn2kqQuz8rBp8VGa1CQiJzgmHMU9R5oGCBO2aBHApQyZd+N4MyeVa+13jFN5LB0I0pGFIwFDWmEGCOBgnYqu82nr2A+1+ArYg5IkB2j+l1CBFKxMEqKa5WZAwvhPdt12f+QNBohjVnels4FmM3YusQJiXRW3HIWSwK7oYjTpCR4Kj4XaghQYI6hwJNyWIEwXepP9uS0sbex7j9a3j/eHnrxbBfF5y7zn3HdXznibPlvHFazq6DKz8qvyq/K3+qn6tfql+r3wro/Nw5o5TGtXvfwFvtGeb</latexit> <latexit sha1_base64="5MA9J47fV/N83fC3Lg9LbvGCH4=">AHpHiclVRtb9MwEA6jUCivg0+IL4ZpLGFsNIAEAk1CgBDSGAzBXtDcVa7rtBZ2EtnOWGVF4tfwFf4O/wa/tFvTdqBFanM53z3Pc3e2OzmjUjWbf87Mna2dO1+/cLFx6fKVq9euz9/YlkhMNnCGcvEbgdJwmhKthRVjOzmgiDeYWSn8+21Xd85IELSLP2iBjlpcdRLaUIxUsbVnq/daizCTUlD2OH6e/kA2PegjMAagLgkFOlWxrvhaX+3pjOS5tAFR9olAJIO5myqUk5RHCoM3tn+YrsfniUWNxM/SokNMu8GOIBEIa0gO83CWhKjUFQnOvWRIaQogR6qPEdNfR7xReQLQkU6dZH5WPR0QeH6ShyF62AZvI2Awe4j5eWVpcVEosfRYfvIBVnWA6MmgPEuNBZnhY+kDoZSbtWHMqpG3KMtXQMNovVIp+iA2uyzcNxt/HBMby2jp9vGNTprvsV1/sZ8RMqZrBM5pks0wsbHEKG0h4jDyUznBplmt5msuAVdlmcUHCmBG64gOpDfxQGea+DmlUGan3vAD+iESNiWzwn+Pj0e98l9O/wnDN7OfHv0wzW8ARhIVjrE60pM5zXGBgvb6KmpfX2iuNt0Dpo14aCwEw2ezPT8nYDfDBSepwgxJuRc3c9XSCiKGSkbsJAkR/gb6pE9Y6aIE9nS7hIrwaLxdEGSCfNLFXDe8QyNuJQD3jGRtmA5uWads9b2CpU8a2ma5oUiKfZEScGAyoC9EUGXCoIVGxgDYUGNVoD7yFxNytybZg+M0fQJOyCqWgjmLS0Tx16R1OFlNfnQF9qAgqTkO84R2n3voYJ4pQNuiRBVP2zktG9qx+Pege0FwOW3cEyYiCmRkbTRGzA3dTr7rdUP1oG/ANMQMSZMOo/pgTgVQmjBJ/JZRmYD14x56Y8l+RND2KNGa1LO0EmGJsX7KcpLr025RlksBOT2RFXhE8le+EGgCUmDH4eFJN8xFml8aTe3La2H60Gj9efTpycLV8P9eiG4HdwNwiAOngYvg3fBZrAV4NqP2s/ar9rv+r36+/rn+pYPnTszLkZVJ76/l85mKrY</latexit> Learning in CRFs ■ As with logistic regression, weights θ are learned by minimizing negative log likelihood: N log P ( y ( i ) | w ( i ) ; ✓ ) X ` = − i =1 sum over all possible labelings N ⇣ ⌘ X X θ · f ( w ( i ) , y ( i ) ) + log θ · f ( w ( i ) , y 0 ) = − exp i =1 y 0 ∈ Y ( w ( i ) ) ■ Can be computed efficiently using forward algorithm . 8

  33. Learning in CRFs 9

  34. Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . 9

  35. <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m X X α m ( y m ) = exp s n ( y n , y n − 1 ) n =1 y 1: m − 1 9

  36. <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 9

  37. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 9

  38. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) y m − 1 9

  39. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) y m − 1 sum instead of max 9

  40. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 Viterbi is a special case of the max-product algorithm , X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) forward is a special case of the sum-product algorithm . y m − 1 sum instead of max 9

  41. <latexit sha1_base64="1jW+4tvq9pLDCdP9LJYNMcCYaQ=">AKNXicrVb9s2Fazra69W7M9oVbERamtRKB3RYSC7YkCRLQOWtkPoCrRE2WxJSiCpNIagP7Q/sN+yh70Ne91fGC+yK9lylgATYPv48JzvfOeiw0lOiVTD4R+3t56+53bvTv9wbvf/Bh3e3P3oqs0LE+CzOaCaeT5DElHB8poi+HkuMGITip9NXn1jzp9dYCFJxn9R8xyPGZpykpIYKa2Ktm/NtiFp5L4cMLK19V9YH7nVQBGAMqCQUoYUTIq2SisXpQn+2FlDKCaYUqAOMkU9YlrZYI84iZr5IdhPofCwa7p75DhYwkwJnZAKlAcQnxZe53UQiqskXBqvd0UMIBZEjNYkTLXxdxg2oD0J5BunGSecN6PSH/yUEY+E/APvg+ABp7hpSjV1UGE4kpQ5fRUgVpNgWLIoBmFQa7XeYLqvOaqinXgUW5cUHeYO29AeuKapBvUIGRjJjfVGsdbOBFZfjliUZdr7o7sbXvsF9h0RFl1U976VoYx9SxKcUP5BQWMG6mVj767E0WDtaVyIKdVED5whMY/tpr5ovRJ0Gqp0zwG7hWxfFvu4D/eH+e/KJb7ZxPY0H3d/PXe125uAihOld+IaoNujqnfFyjIdKZM/ojmM+TqsnhbmlVktomiD3gJkMGZMS1A7eF5MumdbrmIksWfgany/dqEuT6SCPnWbdXJ+/MVroerDM9auNbu01srxlEzwdhWI6Oav1G4PnVmOCqUJmCKuFbFsYvbRrhuo1nyxVZkac/bdeRq97ByqYKy3yabDenSuSeF/DXj1NQJG0d2d4eHQPmBdCGthx6uf02h7S8AkiwuGuYopkvI8HOZqXCKhSExNYCFxDmKX6EpPtciR7ol49LeqhXY1ZoEpJnQH6A1TY9SsSknLOJtjQvoFw9M8qus/NCpV+MS8LzQmEeu0BpQYHKgLmiQUIEjhWdawHFgmiuIJ4h3Q2lL3Jdp0aYGaYXWLUTidm4lKmN3qI0YVXb+dIlOoACc/w6zhDPmshClihM4TnKCKnMJpwu5q173kwuSy7p0S0iKFcz0GiEcUbOA7BZq+2ScatmAL/FukECn2jWP+VYIJUJzcTdUZVu2BR+YlZ4dZUl4UtLbTKi0BnYypS5ZjXlZubdJMYjiZiqzIW4TX/C1RDYBS3QZnj9tuzkJPabg6k+vC06PD8OHh0c+f7x/Xc/rHe+e96ne6H3yDv2fvBOvTMv7m3HvWOe1/1f+/2f+r/7cz3bpV+3zstZ7+P/8CrWyU6g=</latexit> <latexit sha1_base64="S+C2pUnFb9NeQcjrf+w7RtILE54=">AJH3iclVXdbts2FazJfW8v2a73A27IjcNJmVDtjQIkCxDcOAIlsGLG2H0BVoibKJkpRAUmkMQW/RF9jT7G7YbZ9lNzskZceylS0RYJs+PN/3nR8ecVxwps1w+O7Oxnvb27d7X3Q/Cjz/59N72Z891XqEniU5z9XLMdGUM0nPDOcviwUJWLM6Yvx6+/t/osLqjTL5W9mVtCRIBPJMpYQA6Z4e/Ntfxefahbisaje1A+R/Z3VA3SMsC4F5kwo+NKHEf1q+pkP6qtAzZTakiNcJLmxkGyesEwi4X9qsRBP/EoL97GnpWLFiKvJsTyBRJKkwvi7ArhEFdtUJw5j0QZRJhQcw0Ibz6fa47qK8h2rNMt06yWPJeTyh8dhANwmdoH/04QMA9JcaHV9eWk6iJIJfxwoR5PkHzIqDlKvR3u9znoc6aUG25DhzLrQtyxbV3RdalaplvUYFjHYtw2Qw2vMQXV9HjE2Bdr7rfcbXv8F+JokNlFQcoqIV1DjEncsLpVxort3Awq7W/rgVkbUuLUw5h0APvCOzj+3mvmqCtmg1VJveYL8iLh4W3D0P/Pj8fNi+X8ugWu6D81f730D8yeA08yES6pO9HpNmBes2GRqbP6EF1Pi6zKflqsqsfCVdGKuA1pMxRIxIA0hVSLprWCS1Uns5xlqcLe7MgbsJ07KFNfyF7fS9sG6wFbwPldI3JDTgfTFCNmuScNVwcvPjezvBw6B60voiaxU7QPKfx9obCaZ6UgkqTcKL1eTQszKgiyrCE07qPS0LkrwmE3oOS0lAfFS5+6NGu2BJUZYr+EiDnHUZURGh9UyMwdMeNb26Z41de+elyb4dVUwWpaEy8UJZyZHJkb2MUMoUTQyfwYIkikGsKJkSuBUMXFnQ+SWZKeUX1LQTScSo0plTb4U0FnUbfOkT7WNFJX2T5EIQmT6ocEYE47OUZqTkxl432XzdVa+H6QUrdFO6BSWnBucwMEwSbkfNzVvb7MbJD1Uf/0ChQYqeQNS/FQRkyuIxL+Na2jYBN+3L6v6vzyZXHjCsp1W5QKAZGxd8oLKqvYvCJ5riscTlZdFK+A1vAsUCEgGbfD+tA3zHnBKo9Uzub54fnQYPTo8+vXrnafNe1F3wRfBmEQR8EzwNfgpOg7Mg2fxn6/7Wg6393h+9P3t/9f72rht3GsznQevpvfsXiMUzJw=</latexit> <latexit sha1_base64="mezX0sqYlDlGzh2M6vW+sE82j1s=">AKNXicrVXbtGEGWcNorUW9w+5mVbwzBZx67oBkiRQIDTGwoEbl2gzgVehViRS2mT3SWxu3QsEPyh/kC/pQ9K/raX+heKIWUqKRGS0DSaHZmzsyZ4ewkp0Sq4fD3a1vX3n3Ru9mf/De+x98+NGt7Y8fy6wQMT6LM5qJpxMkMSUcnymiKH6aC4zYhOInk5fmPMnF1hIkvFf1DzHY4amnKQkRkqrou0bvw524akPpyw8lV1B5jfeRWAEYCyYJASRpSMSjYKq+flyX5YGQOoZlihCsA4yZR1SatlhHnEzFfJDkL9jwWD3VPfRYWMJMCZWYBUoLiE+DL3u1IqrKVglXvaVDCAWRIzWJEy2cL3KDaEGjPRLpykXnDer0g/9FBGPiPwD74PgA69gwpl15VmZhITBm6jJYqSLMpWJAmiwMdrvMF6nO61QNXQc2ypUJeR1r73WwLlQT+QoMjGTE/KZa62AjXlSG90901HXW3YnlvsN+JYsOlFU/7aW5MY+pIhPKf5CQmEF62aw9texdLA2WhcWxJTqRA+cITGP7a+bz0SdBqdM8AO4Vsfm23MFb3h/nvyDL/bMFbOi+bv5672s3NwEUp8pvoFrQzZj6fYGCTGfK1I9oPkOl8Xb0mSRWRYNiD3gpkIGZMS1A7dEckvkJs9cZMnCzYTpcH1LDuQKkUbOtW6vLt7ZrXQ9WAc4agNYu/8IoueDMCxBXZ3V+o3Bc6sxwVQhM4TVQrYtjF7YNUP1mk+WKjMjzTn7jyNXnQOVTDW2TYT06/zKF/xXwzdcIGEW3doaHQ/uAdSGshR2vfk6j7S0BkywuGOYqpkjK83CYq3GJhCIxdUAFhLnKH6JpvhcixzploxLe6tWYFdrEpBmQn+4Alb9CgRk3LOJtrSvIBy9cwou87OC5V+NS4JzwuFeyA0oIClQFzRYOECBwrOtcCigXRuYJ4hnQ3lL7INU8NmBmF1i1C4nZuJSpRW+lNGFV2/nSFTqAnP8Ks4YQz5vIQpYoTOE5yigipzCacLuYuvO8kFyWVN3TIkxQpmeo0QjqhZQHYLtdV2ybhVM4DfYt0gU901j/lWCVCZ2Ju6Mq3bAp/NSs8OpNloQvLbXYLqu0CehiDC9ZjnlZubVJM4nhZCqyIm8lvOZvE9UBUKrb4Ox281Z6CkNV2dyXh8dBh+eXj0892d46/reb3p3fY+83wv9O5x94P3ql35sW97d693nHvYf+3/h/9P/t/OdOta7XPJ17r6f/9D9AplSI=</latexit> <latexit sha1_base64="KhLME4V4Sic9fp1LltTVpr0QY=">AKNXicrVXdbuNEFPZ2YbMJf9vlkpuBqpNtyUuSItAkcqvkFaFItHdRZ2sNbHyezO2NbMpNvI8gvxAjwLF9whbnkFzsw4WTtxCxVYSnJy5nznO38+Myk4U3o4/O3W1u3Xr/Tu9sfvPHmW2+/c2/7/mOVz2VMz+Kc5/LphCjKWUbPNOcPi0kJWLC6ZPJi6/M+ZMLKhXLs5/0oqBjQaYZS1lMNKi7Tu/DHbxqWI+nojyZfUAmd9FaARwmouMGeCaRWVYhRWz8qT/bAyBljPqCYVwnGSawtJq5WHRSTMVykOQvgngsHuqe+8YsES5MwsQSpJXGJ6WfhdIQRV2QrBqveAlGUIC6JnMeHlz0veoLrC0Z7xdOMki4b1ZkL+o4Mw8B+hfRtgMD3jGgXlUZn0ROBbmMVirM8ylaFgE1qzDY7TJfhrqoQzXlOrBeblyQV72XjnrYjWeb1CBkYqE31SDjf8RWX42Ql43ay6O7G17Bfi6KDZR0HKiFMfYxJ9mU048UlawMO1v8kFztpsXVyYcg6BHjhDZgy/bzXzWemzoNVSp/kcuVfExtuCo394fx+WSz3zyZwRfeh+Zu9r2FuAjhNtd9gtaRXc8L7giWbznQwIQXM+LKsnxZmkUtoiGwx5kJkGBVJQBILN1zFY9a0FZDS1knixk8X9vogbuJp5KB1eyF5Z7fW9WCT4KhNYO3+IwnMBxNUoTo7q/Ubg+dWY0K5JmYIq6VsWxg9t2uGw5pPViozI805+Y8jZ53DlUwhm1y1WE9Ov8yhP+V8PprBI2iezvDw6F90KYQ1sKOVz+n0faWxEkezwXNdMyJUufhsNDjkjNYk6rAZ4rWpD4BZnScxAzAi0Zl/ZWrdAuaBKU5hI+mUZW20SURCi1EBOwNC+gWj8zyq6z87lOPx2XLCvmaxI0rnHOkcmSsaJUzSWPMFCSWDGJF8YxANzRc5FCnBs2M8guq24nEYlyq1LK3QpqIqg2+dIkOsKQZfRnQpAs+bDEKRGMLxKakjnX5hJOl3JXvR4kF6xQdelWLjnVOIc1wjLCzQKyW6itkvGrZoB/pCgyQ9gah/KgkOpcQibujKmjYFL9vVnh1nSXLVpYgtMqbQCQjKlLXtCsrNza5LmieDKV+bxoBbyBt4GCA5JCG5w9bcOcBUxpuD6Tm8Ljo8Pw48OjHz/ZOf6ynte73nveB57vhd5D79j7zjv1zry4t9172DvufdH/tf97/4/+n85061aNedrPf2/gYLQpVa</latexit> Learning in CRFs ■ Likelihood can be computed efficiently using forward algorithm . Define: m m X X X Y α m ( y m ) = exp s n ( y n , y n − 1 ) = exp s n ( y n , y n − 1 ) n =1 n =1 y 1: m − 1 y 1: m − 1 ■ As in decoding / Viterbi, can be decomposed into recursive substructure: m − 1 X X Y = (exp s m ( y m , y m − 1 )) exp s n ( y n , y n − 1 ) n =1 y m − 1 y 1: m − 2 Viterbi is a special case of the max-product algorithm , X = (exp s m ( y m , y m − 1 )) × α m − 1 ( y m − 1 ) forward is a special case of the sum-product algorithm . y m − 1 sum instead of max M s m ( k, k 0 ) ⊗ v m � 1 ( k 0 ) . v m ( k ) = k 0 2 Y 9

  42. Learning in CRFs 10

  43. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: N �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 10

  44. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 10

  45. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! 10

  46. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. 10

  47. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) 10

  48. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k forward score: α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  49. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k backward score: forward score: sum over all suffixes α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  50. <latexit sha1_base64="gBH4cwI4J7/lJfldZ9/VtEYFLGQ=">AKFHicrVfbxtFEL+mkNTmT5si9YUHFqIoZ9KkvoBEVRSp4p+QqkCQSFuUdU/ruz1729290+46jXW6D8AX4NPwhnjlnW/BR2B29+z47AskgpNsj2dn5jfzm9m5YcGZNv3+nzfWbr7x5vrGrU73rbfef2nc27T3U+UQk9SXKeq+dDoilnkp4YZjh9XihKxJDTZ8NX9rzZ2dUaZbLH820oANBRpJlLCEGVPHm+s/dbXysWYiHonxd3Uf2d1r10CHCeiIwZ4IZHZfiMKpelEe7UWUNsBlTQyqEkzQ3ziWr5hGmsbBfpdiL4J/odbePQx8VC5Yib+YAMkWSEtPzImxLoVeVjRScegdAmURYEDNOC9/muH2qksC7dhI1y6yWLBeLSh8shf1widoF3TQxB7TIxPr6psTKJGgpzHcxXm+QjNSECLHS328xnqU7rVC1dey7KtQm5iLVzEawN1Ua+BgOHOhbhohp0eCFeXEaPjiDqKuv+xHfYr+URQvKsh94ARfWOMScyBGnDzRWTnBuFmt3FQuCNdHasDlHBLd84bMGn7XaOaLMmS9Rku95nPkr4jLt+GO/uX+eP8ZWf6fK+CS7kPzV3tfu/kJ4DQz4QKqA70cE+4LVmw0NrZ+wosx8bzMbsFi+yRcCxaEHcgbYUC6ViCg3REynTWl0LlaczPxunzfdqSVwl0qF3rfsL1Xu7pb3VgEOmgDO7j+CwIAwQTWq3PacGHy/G5MKTfETmE1k10P45duz3DY8+lcZYdkcdC+Ps3il61T1RvAOrnsJ6d7pUy+B/xuvGdrf5+3z1oVYhqYSuon+N4c03hNE8mgkqTcKL1adQvzKAkyrCE06qLJ5oWJHlFRvQUREmA8EHpXpoV2gZNirJcwUca5LSLHiURWk/FECzt/dLZ1bZdnY6MdnDQclkMTFUJh4om3BkcmTfwChliaGT0EgiWKQK0rGBMg28J6Gzi/AjCk/o6ZSCIGpc4ceiOloaiazue+0C5WVNLXS4EkenHJc6IYHya0oxMuLHv2Gwmt/F1Pz1jha6pm4fk1OActgSThNv94pZMU+12iN8kXfwVhQYpegRZf19QRUyuIBP/CqgYSP8od3Q1T9ZMjm3BLFZVukSgGIsL3lBZVn5rchzTfFwpPJ0Uh4xd8lCgFIBm3w9rTp5i1gSqPlmVwVnh7sR5/sH/zw6dbjL+p5vRW8H3wUhEUfBY8Dr4NjoOTIFn/a+PexgcbqPNL59fOb53fvenajdrnvaDxdP74G06/io=</latexit> Learning in CRFs ■ As in logistic regression, gradient of the likelihood w.r.t. parameters is difference between observed and expected feature counts: count of feature j for token sequence w (i) , N tag sequence y (i) �` X E [ f j ( w ( i ) , y )] − f j ( w ( i ) , y ( i ) ) = �✓ j i =1 ■ Gradients can be computed by automatic differentiation! ■ In the Olden Days, would use the forward-backward algorithm to compute expected counts. Y m � 1 = k 0 Y m = k transition score backward score: forward score: sum over all suffixes α m � 1 ( k 0 ) exp s m ( k, k 0 ) β m ( k ) sum over all prefixes 10

  51. Better features for sequence labeling? ■ Until now: hand-engineered features: 11

  52. Better features for sequence labeling? ■ Until now: hand-engineered features: w i contains a particular prefix (from all prefixes of length  4) w i contains a particular suffix (from all suffixes of length  4) f i ±{ 0 , 1 , 2 , 3 } , ( m i − 2 ,i − 1 ), ( m i − 1 ,i ), ( m i − 1 ,i +1 ), w i contains a number ( m i,i +1 ), ( m i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i ), ( m i − 1 ,i,i +1 ), Lexical w i contains an upper-case letter ( m i,i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i +1 ), ( m i − 1 ,i +1 ,i +2 ) w i contains a hyphen p i − { 3 , 2 , 1 } , a i + { 0 , 1 , 2 , 3 } , ( p i − 2 ,i − 1 ), ( a i +1 ,i +2 ), ( p i − 1 , a i +1 ), ( p i − 2 , p i − 1 , a i ), ( p i − 2 , p i − 1 , a i +1 ), POS w i is all upper case ( p i − 1 , a i , a i +1 ), ( p i − 1 , a i +1 , a i +2 ) w i ’s word shape Affix c :1 , c :2 , c :3 , c n : , c n − 1: , c n − 2: , c n − 3: w i ’s short word shape initial uppercase, all uppercase/lowercase, w i is upper case and has a digit and a dash (like CFC-12 ) Binary contains 1/2+ capital(s) not at the beginning, w i is upper case and followed within 3 words by Co., Inc., etc. contains a (period/number/hyphen) 11

  53. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: w i contains a particular prefix (from all prefixes of length  4) w i contains a particular suffix (from all suffixes of length  4) f i ±{ 0 , 1 , 2 , 3 } , ( m i − 2 ,i − 1 ), ( m i − 1 ,i ), ( m i − 1 ,i +1 ), w i contains a number ( m i,i +1 ), ( m i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i ), ( m i − 1 ,i,i +1 ), Lexical w i contains an upper-case letter ( m i,i +1 ,i +2 ), ( m i − 2 ,i − 1 ,i +1 ), ( m i − 1 ,i +1 ,i +2 ) w i contains a hyphen p i − { 3 , 2 , 1 } , a i + { 0 , 1 , 2 , 3 } , ( p i − 2 ,i − 1 ), ( a i +1 ,i +2 ), ( p i − 1 , a i +1 ), ( p i − 2 , p i − 1 , a i ), ( p i − 2 , p i − 1 , a i +1 ), POS w i is all upper case ( p i − 1 , a i , a i +1 ), ( p i − 1 , a i +1 , a i +2 ) w i ’s word shape Affix c :1 , c :2 , c :3 , c n : , c n − 1: , c n − 2: , c n − 3: w i ’s short word shape initial uppercase, all uppercase/lowercase, w i is upper case and has a digit and a dash (like CFC-12 ) Binary contains 1/2+ capital(s) not at the beginning, w i is upper case and followed within 3 words by Co., Inc., etc. contains a (period/number/hyphen) f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 11

  54. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  55. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: ■ interpretable, explainable ■ can generalize well ■ fast training and inference ■ channel domain knowledge f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  56. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Better features for sequence labeling? ■ Until now: hand-engineered features: pros: cons: ■ interpretable, explainable ■ can be sparse/high variance ■ can generalize well ■ lack of shared representations ■ fast training and inference ■ task-specific ■ channel domain knowledge ■ worse performance f ( w , y m , y m − 1 , m ) = 1 0 … 1 … 0 … 1 0 0 0 0 0 … 0 w m = back w m+1 = ache w m-1 = will w m-1 = my y m-1 = MD Janet will back the bill . 12

  57. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. f ( w , y m , y m − 1 , m ) = . . . . . . 13

  58. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: f ( w , y m , y m − 1 , m ) = . . . . . . 13

  59. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: ■ shared representations ■ channel external knowledge (e.g. word embeddings) ■ high accuracy f ( w , y m , y m − 1 , m ) = . . . . . . 13

  60. <latexit sha1_base64="y2cxWKwz3S3xVgA5ofaWNPquJR0=">AKNXicrVXdbts2FbTra69v6a73A23Ii8NJmVDejQwUD2iwFtgxY2g6hK9ASZbMlKYGk0xiCXmgvsGfZxe6G3e4VdkjKjhUra4JNgO3jw3O+7/zpcFxwps1g8PutjdtvHmnc7fbe+vtd959797m/Sc6n6mEniQ5z9WzMdGUM0lPDOcPisUJWLM6dPxy6/t+dMzqjTL5c9mXtCRIBPJMpYQA6p486vW18rFmIx6J8VT1A9nde9dEQYT0TmDPBjI5LMYyq5+XRblRZA2ym1JAK4STNjXPJqiXCPBb2qxR7EfwT/d72cehRsWAp8maOIFMkKTE9L8K2EPpV2QjBqXeAlEmEBTHThPDylwVv7oCaMci3TjJYsV6PaHw8V7UDx+jXfRdHwH2lBgfXlVZTKImgpzHSxXm+QtioBWq9DbjNfhDqvQ7Xl2nMoNy7IBdbOBVgbq0W+QWGOhbhqhp0eAUvLqNHR4C6XnV/4mrfYn8pihaWy37gBbWwxiHmRE4/URj5QTnZrl217kArMnWxoUp5xDonjdk1vCHRjOflyHrN1rqNV8g/4q4eBvu6DXvj/dfFMv/cwlc0X1o/nrvazc/AZxmJlxhdaRXc8L7ghWbTI3Nn/BiSnxdFm/LRXZI+GqaEncgbQZCqRjCQ7SFVIum9bqWqg8XfhZnDbf6wVxHaShd637C9l7u0t768THDQJnN1/JIEBYJqVGfntOHK5PndmFJuiJ3CaiG7HsYv3J7hsOfTpcoOyeqgfXuaxS9ap6o/gnVy1WE9O9cM4X8kfM01gobxva3B/sA9aF2IamErqJ/jeHND4TRPZoJKk3Ci9Wk0KMyoJMqwhNOqh2eaFiR5Sb0FERJoCOj0t2qFdoGTYqyXMFHGuS0qx4lEVrPxRgs7QuoL59ZdvZ6cxkn49KJouZoTLxRNmMI5Mje0WjlCmaGD4HgSKQawomRJohoGLHPqyQjOl/IyaZiKJGJU6c+yNkMaiajqf+0R7WFJXyW5ESmH5c4I4LxeUozMuPGXsLZQm6r14P0jBW6Lt0SklODc1gjTBJuF5DbQk21WzJ+1fTwNxQapOgRP1jQRUxuYJI/B1VQcMm+EO7wqt/s2RyaQliM63SBQDJ2LrkBZVl5dcmzXF4nKZ0Uj4DV/FygAkAza4O1p081bwJRGl2dyXhysB9un/w02dbh1/V83o3+CD4KAiDKHgYHAbfB8fBSZB0NjsPO4edL7u/df/o/tn9y5tu3Kp93g8aT/fvfwDACJWS</latexit> Neural sequence labeling ■ Parameterize f with a (deep) neural network. cons: pros: ■ hard to interpret feature ■ shared representations meaning, explain predictions ■ channel external knowledge (e.g. ■ optimization/hyperparameters word embeddings) ■ prone to overfitting ■ high accuracy ■ compute-heavy f ( w , y m , y m − 1 , m ) = . . . . . . 13

  61. Neural sequence labeling 14

  62. Neural sequence labeling word embeddings will </s> Janet <s> back the bill 14

  63. Neural sequence labeling neural network word embeddings will </s> Janet <s> back the bill 14

  64. Neural sequence labeling per-token features neural network word embeddings will </s> Janet <s> back the bill 14

  65. Neural sequence labeling VB? DT? NNP MD NN? per-token features neural network word embeddings will </s> Janet <s> back the bill 14

  66. Neural sequence labeling Bidirectional RNNs word embeddings will </s> Janet <s> back the bill 15

  67. Neural sequence labeling Bidirectional RNNs forward RNN word embeddings will </s> Janet <s> back the bill 15

  68. Neural sequence labeling Bidirectional RNNs forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  69. Neural sequence labeling Bidirectional RNNs per-token features concatenate forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  70. <latexit sha1_base64="/4jVRd01VWx7s0GMy/AWIkSfBo=">AD43icfVJb9MwFHabAaPcOnjkJVBV2hBUbTcJIQ0AQ+8Ia0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u5HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2B7FQLZ1RiBZnuzj817M3ara6nW65wk3QW4EWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ5BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ=</latexit> <latexit sha1_base64="/4jVRd01VWx7s0GMy/AWIkSfBo=">AD43icfVJb9MwFHabAaPcOnjkJVBV2hBUbTcJIQ0AQ+8Ia0bpPqUDnuSWvNl8h2SouVX8Ab4pV/xQt/hSectEXLOnAU+8u5HznO45Tzoztdn/V6sHWtes3tm82bt2+c/dec+f+iVGZpjCgit9FhMDnEkYWGY5nKUaiIg5nMbnbwr/6Qy0YUoe20UKkSATyRJGifWmUVO2cSzcPA9fhcP5yLFnB/nLsAT7a9Bfg14eNdr4eAqWfHK7c2xV+GUvb2B7FQLZ1RiBZnuzj817M3ara6nW65wk3QW4EWq2j0U49wmNFMwHSUk6MGfa6qY0c0ZRDr56ZiAl9JxMYOihJAJM5Eph8rDtLeMwUdq/0oal9WKGI8KYhYh9ZNGEuewrjFf5hplNXkSOyTSzIOmyUJLx0EtSqByOmQZq+cIDQjXzXEM6JZpQ62fhRbxQZgp8BrbaCBWRM0lZvUIpFv5bg4TPVAlB5PiJwkRjC/GkJCM29xhk6zxVdI8Hc9YalYqzZcyNTAHi5VmEyYJ5BYXGxVsz+mFpd7A78FPwsN7z3BDyloYpX2TIieFP3s5ngR7iA/4tk8m+kh9W2XEnAN1NIoFKQLi8h5coAjidaZWmF8EZ+SdT/gCRe8WU8VNOWEf5C9i5fv01w0u/09jv9jwetw9erq7mNHqLHaBf10HN0iN6hIzRAFP1Ev2tBbSuA4GvwLfi+DK3XVjkPUGUFP/4A1gxOkQ=</latexit> Neural sequence labeling Bidirectional RNNs D J N B T P H D M D N R V W A P softmax( ) per-token features concatenate forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 15

  71. Neural sequence labeling Bidirectional RNNs VB? NNP MD per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 16

  72. Neural sequence labeling Bidirectional RNNs t i-1 VB? NNP MD per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 16

  73. Neural sequence labeling Bidirectional RNNs t i-1 VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 17

  74. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 17

  75. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 18

  76. Neural sequence labeling Bidirectional RNNs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 18

  77. Neural sequence labeling Bidirectional RNN-CRFs VB? DT? NNP MD NN? per-token features forward RNN backward RNN word embeddings will </s> Janet <s> back the bill 19

  78. Neural sequence labeling VB? DT? NNP MD NN? per-token features neural network word embeddings will </s> Janet <s> back the bill 20

  79. Convolutional neural networks 21

  80. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. 21

  81. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. 21

  82. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations. wait for the video and do n't rent it n x k representation of Convolutional layer with Fully connected layer Max-over-time sentence with static and multiple filter widths and with dropout and pooling non-static channels feature maps softmax output Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014. 21

  83. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. ■ Unlike computer vision, in NLP we use 1D CNNs. ■ For sentence/document classification: pooling function over representations. ■ For example: sum, average. Most common: max pooling (over time). wait for the video and do n't rent it n x k representation of Convolutional layer with Fully connected layer Max-over-time sentence with static and multiple filter widths and with dropout and pooling non-static channels feature maps softmax output Figure from: Yoon Kim. Convolutional Neural Networks for Sentence Classification. EMNLP 2014. 21

  84. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d word ] will Janet <s> back the bill 22

  85. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d word ] will Janet <s> back the bill kernel size = 3 22

  86. Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ kd word ] [ d word ] will Janet <s> back the bill kernel size = 3 22

  87. <latexit sha1_base64="oC6lR82ch2B7kQhLCX1aCX53M=">ADtXicfVLbtNAEN3GXIq5pfDIiyGKVBEcVqpSAgpAh54QRSpaStlTbRej5NV92LtrkOC5d/ha3gFib9h7SobipG8vp4Lp4zZyfODO23/+z0/Ju3Lx1e/eOf/fe/QcP23uPTo3KNYURVzp85gY4EzCyDL4TzTQETM4Sy+eF/Fz+agDVPyxC4ziASZSpYySqxzTdrDLo5FsSiDt8F4MSnYq8PyTVCDgw0YbEBYRj4+mYElX4v9BbYq+P68nLQ7/V6/tmAbhGvQWs7nuy1IpwomguQlnJizDjsZzYqiLaMcih9nBvICL0gUxg7KIkAExX1qGXQdZ4kSJV2j7RB7b1cURBhzFLELlMQOzNXY5Xzutg4t+nrqGAy1IumqU5jxwU1a6BQnTQC1fOkCoZo5rQGdE2qdun73cpsZ8DnY5iBURIVJ6+4NSrFw3xokfKNKCKTFwVOiWB8mUBKcm7LApt0g6+T5mUyZ5lZq7RYyeRjDhYrzaZMEs4htbg6m73mlcnz7+AO4uNHxyBD9noIlV2jEheirIonR3M8VPcQX/l8nkv0wHm2MVNQE3TCWBykAWZQ0pVwZwPNUqzxqEt+prou4HJHWKr/KhWbKcAsZXl2/bXA6IUHvcGXw87w3Xo1d9ET9AztoxAdoSH6iI7RCFH0A/1Ev9Bv78iLvMRLV6mtnXNY9QwT/0FBYlBGA=</latexit> Convolutional neural networks ■ In NLP , CNNs merge information across contiguous, fixed-width spans of tokens. dims: [ d z ] # filters [ kd word x d z ] Θ ( x → z ) [ kd word ] [ d word ] will Janet <s> back the bill kernel size = 3 22

Recommend

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For

Greedy Algorithms Chapter 16 1 CPTR 430 Algorithms Greedy Algorithms Greedy Algorithms For some optimization problems, dynamic programming algorithms are overkill A greedy algorithm always makes a choice that looks best at the moment

1.09k views • 65 slides
Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms

Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms

Evolutionary Algorithms CS 478 - Evolutionary Algorithms 1 Evolutionary Computation/Algorithms Genetic Algorithms l Simulate natural evolution of structures via selection and reproduction, based on performance (fitness) l Type of

524 views • 30 slides
Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms?

Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms?

Graph Algorithms Chapter 22 1 CPTR 430 Algorithms Graph Algorithms Why Study Graph Algorithms? Mathematical graphs seem to be relatively specialized and abstract Why spend so much time and effort on algorithms for such an obscure area?

744 views • 47 slides
- - packing p a - packing algo- packing cking rithms algo- a l g o - theorems rithms

- - packing p a - packing algo- packing cking rithms algo- a l g o - theorems rithms

algorithmic composition dForm + IM WS2010 Inst. f. Arch. a. Media packing algorithms packing algorithms packing algorithms p a c k i n g packing algorithms packing algorithms algorithms packing packing algorithm 1 algorithms - -

39 views • 3 slides
Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms

Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms

Algorithms Chapter 3 Chapter Summary Algorithms n Example Algorithms n Algorithmic Paradigms Growth of Functions n Big- O and other Notation Complexity of Algorithms Algorithms Section 3.1 Section Summary Properties of Algorithms

847 views • 83 slides
Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander

Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander

Algorithms Theory Algorithms Theory 10 10 Greedy Algorithms G d Al ith Dr. Alexander Souza Winter term 11/12 Greedy algorithms 1 1. Introductory remarks Introductory remarks 2. Basic examples: The coin-changing problem

504 views • 19 slides
Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of

Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of

Randomized Algorithms Randomized Algorithms Two Types of Randomized Algorithms Two Types of Randomized Algorithms and and Some Complexity Classes Some Complexity Classes Speaker: Chuang-Chieh Lin Advisor: Professor Maw-Shang Chang National

910 views • 52 slides
The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in

The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in

The Role of Algorithms in Computing Chapter 1 1 CPTR 430 Algorithms The Role of Algorithms in Computing What is an Algorithm? Any well-defined computational procedure that takes some value (or set of values)its input and produces some

321 views • 13 slides
Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for

Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for

Algorithms X. Zhang Fordham Univ. 1 Real World applications of algorithms Algorithms for solving specific, complex, real world problems: Google's success is largely due to its PageRank algorithm, which determines importance&quot;

1.03k views • 46 slides
Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for

Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for

Mathematical Analysis of Genetic Algorithms Genetic Algorithms are not appropriate for certain problems where finding the exact global optimum is required. Genetic Algorithms are non-deterministic. However, there are theories about

271 views • 5 slides
Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper

Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper

Jesse Anderson CSCI 446, Artificial Intelligence Michelle Van Dyne 11-16-2016 Genetic Algorithms Welcome to this introduction to Genetic Algorithms. The objective of this paper is to inform you, the reader, on Genetic Algorithms and give

564 views • 9 slides
Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest

Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest

Efficient Algorithms P and NP So far, we have developed algorithms for finding shortest paths in graphs, CISC5835, Algorithms for Big Data minimum spanning trees in graphs, CIS, Fordham Univ. matchings in bipartite graphs,

514 views • 8 slides
Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph

Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph

Algorithms in Nature Nature inspired algorithms http://www.cs.cmu.edu/~02317/ Ziv Bar-Joseph Matt Ruffalo zivbj@cs.cmu.edu mruffalo@andrew.cmu.edu GHC 7715 GHC 8006 Topics Introduction (1 Week) Classic algorithms (4 weeks)

1.68k views • 28 slides
Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline What is algorithm: word origin, first algorithms, algorithms of todays world Sequential algorithms, Parallel algorithms, approximation

828 views • 62 slides
Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline

Algorithms for Big Data CISC5835 Fordham Univ. Instructor: X. Zhang Lecture 1 Outline What is algorithm: word origin, first algorithms, algorithms of todays world Sequential algorithms, Parallel algorithms, approximation

332 views • 21 slides
Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion

Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion

Algorithms and Architecture 1 Introduction to Algorithms Alexandre David 1 Outline Notion of algorithms, GCD example. Algorithmic problem solving. Problem types. Sorting example. Numerical example. Analyzing algorithms.

333 views • 12 slides
GENETIC ALGORITHMS By Joy Reistad Overview What are genetic algorithms? History

GENETIC ALGORITHMS By Joy Reistad Overview What are genetic algorithms? History

GENETIC ALGORITHMS By Joy Reistad Overview What are genetic algorithms? History Methodology Initialization Selection Crossover Mutation Examples What are genetic algorithms? Type of search used in artificial

670 views • 33 slides
Chapter 7 Approximation Algorithms CS 573: Algorithms, Fall 2013 September 17, 2013 7.0.0.1

Chapter 7 Approximation Algorithms CS 573: Algorithms, Fall 2013 September 17, 2013 7.0.0.1

Chapter 7 Approximation Algorithms CS 573: Algorithms, Fall 2013 September 17, 2013 7.0.0.1 Todays Lecture Dont give up on NP-Hard problems: (A) Faster exponential time algorithms: n O ( n ) , 3 n , 2 n , etc. (B) Fixed parameter

583 views • 13 slides
Algorithms for prerequisites Computer Games fundamentals of algorithms and data structures

Algorithms for prerequisites Computer Games fundamentals of algorithms and data structures

Course syllabus credits: 2 cu Algorithms for prerequisites Computer Games fundamentals of algorithms and data structures (see Cormen et al., Introduction to Algorithms ) knowledge in programming (e.g., with Java) Jouni Smed

406 views • 5 slides
Visualisatie BMT Algorithms - 2 Arjan Kok a.j.f.kok@tue.nl 1 Lecture overview Vector

Visualisatie BMT Algorithms - 2 Arjan Kok a.j.f.kok@tue.nl 1 Lecture overview Vector

Visualisatie BMT Algorithms - 2 Arjan Kok a.j.f.kok@tue.nl 1 Lecture overview Vector algorithms Tensor algorithms Modeling algorithms 2 Vector algorithms Vector 2 or 3 dimensional representation of direction and

1.15k views • 45 slides
Analysis of Algorithms Amr Magdy Analyzing Algorithms Algorithm Correctness 1. Termination a.

Analysis of Algorithms Amr Magdy Analyzing Algorithms Algorithm Correctness 1. Termination a.

CS141: Intermediate Data Structures and Algorithms Analysis of Algorithms Amr Magdy Analyzing Algorithms Algorithm Correctness 1. Termination a. Produces the correct output for all possible input. b. Algorithm Performance 2. Either

1.02k views • 64 slides
Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13

Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13

TU/e Algorithms (2IL15) Lecture 13 Algorithms (2IL15) Lecture 13 Wrap-up lecture 1 TU/e Algorithms (2IL15) Lecture 13 Part I of the course: Optimization Problems we want to find valid solution minimizing cost (or maximizing

904 views • 20 slides
LP/SDP LP/SDP Algorithms Algorithms Claire Mathieu Brown University Many optimization

LP/SDP LP/SDP Algorithms Algorithms Claire Mathieu Brown University Many optimization

LP/SDP LP/SDP Algorithms Algorithms Claire Mathieu Brown University Many optimization problems can be written as integers programs Traveling salesman tour, vehicle routing Steiner tree, connecting Clustering, cuts Coloring

820 views • 68 slides
More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1

More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1

More Graph Algorithms Data Structures and Algorithms CSE 373 SP 18 - KASEY CHAMPION 1 Announcements Calendar on the webpage has updated office hours for this week. Last Time: Dijkstras Algorithm to find shortest paths. Today: Two more

509 views • 29 slides

More recommend