How Good are Humans at Solving d r CAPTCHAs? A Large Scale - PowerPoint PPT Presentation

Authorize Baidu captcha.net eBay Digg Google http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Baidu captcha.net eBay Digg Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu captcha.net eBay Digg Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu Microsoft captcha.net eBay Digg Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu Microsoft captcha.net recaptcha eBay Digg Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu Microsoft captcha.net recaptcha Skyrock eBay Digg Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu Microsoft captcha.net recaptcha Skyrock eBay Digg Slashdot Google Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Yahoo Baidu Microsoft captcha.net recaptcha Skyrock eBay Digg Slashdot Google mail.ru Blizzard http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Digg eBay Google http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Microsoft Digg eBay Google http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Microsoft Digg recaptcha eBay Google http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Microsoft Digg recaptcha eBay Slashdot Google http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Authorize Microsoft Digg recaptcha eBay Slashdot Google Yahoo http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly 1000 0.1% Precision accuracy http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly 1000 x 3 0.1% Precision Knowing the accuracy probable answer http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly 1000 x 3 3000 0.1% Precision Knowing the by scheme accuracy probable answer http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Precision is costly 63000 captcha http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Underground API http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

MTurk by Amazon Worker(s) Requester (us) http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Requester interface http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Worker interface http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes • 13 images schemes http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk) http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk) • 5000 x 3 image captchas / scheme (MTurk) http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Largest captcha experiment ever • 8 audio schemes • 13 images schemes • 1000 x 3 image captchas / scheme (bypass-captcha) • 3500 x 3 audio captchas / scheme (MTurk) • 5000 x 3 image captchas / scheme (MTurk) • 318 000 captchas annotated overall http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Language repartition Other 149 Russian 12 Balochi 13 Portuguese 15 Hebrew 15 Punjabi 15 Vietnamese 15 Bikol 16 Cebuano 17 Arabic 19 Macedonian 21 Dutch 21 French 23 German 28 Gujarati 30 Slovene 33 Marathi 39 Mandarin 51 Bengali 52 Kannada 64 Spanish 71 Romanian 95 Telugu 331 Hindi/Urdu 578 Malayalam 625 English 2791 Tamil 3502 0 500 1000 1500 2000 2500 3000 3500 4000 http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Repartition by education &"#&"# &'"# !!"# $%"# ()*+,-./0# 123+#4*+..-# 5)06,/# 7.#8./9)-#,:;*)<.=# >+?@# http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Age repartition 800 727 708 686 658 700 647 600 500 Number of users 429 388 364 361 400 351 318 268 300 239 230 205 182 177 200 137 133 132 113 106 104 103 91 76 100 70 63 58 56 46 42 38 37 36 35 26 26 26 21 18 16 15 11 10 6 2 2 2 2 1 1 0 18192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646667687172 http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Nb of user distinct answer for image scheme (bp) 1 answer 2 answer 3 answer 3.8934 Yahoo 25.615 70.492 3.8462 Blizzard 24.519 71.635 19.306 Slashdot 33.839 46.855 1.4778 Skyrock 18.966 79.557 21.729 Recaptcha 40.576 37.694 5.6893 Microsoft 25.821 68.49 37.44 Mail.ru 42.512 20.048 5.3333 Google 24 70.667 2.7311 eBay 16.597 80.672 5.2036 Digg 30.543 64.253 14.139 Captchas.net 42.008 43.852 4.3678 Baidu 21.839 73.793 0.69124 Authorize 13.825 85.484 0% 20% 40% 60% 80% http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Nb of user distinct answer for image scheme (mk) 1 answer 5.2632 2 answer Yahoo 26.032 3 answer 68.705 0.87449 Blizzard 13.632 85.494 5.7506 Slashdot 26.132 68.117 0.89514 Skyrock 11.995 87.11 18.689 Recaptcha 38.565 42.747 13.119 Microsoft 33.633 53.248 24.388 Mail.ru 41.004 34.608 7.8974 Google 25.385 66.718 1.9013 eBay 15.827 82.271 2.4745 Digg 19.031 78.495 7.9385 captchas.net 32.113 59.949 2.7027 Baidu 16.319 80.978 0.23095 Authorize 6.5178 93.251 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Nb of user distinct answer for audio scheme 1 answer 2 answer 29.197 Yahoo 3 answer 36.439 34.363 28.04 Slashdot 39.474 32.486 66.62 Recaptcha 25.512 7.8678 87.65 Microsoft 10.823 1.5264 95.403 Google 4.1875 0.40965 36.417 eBay 38.797 24.787 86.372 Digg 12.615 1.0129 41.655 Authorize 39.232 19.113 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Image solving time authorize 200 baidu 0 captchas.net 200 digg 0 ebay 100 google 0 200 mailru mslive 0 recaptcha 200 skyrock 0 slashdot 100 blizzard 0 yahoo 50 0 100 0 100 0 100 0 200 0 100 0 100 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Audio solving time Authorize 50 Digg 0 eBay 50 Google Microsoft 0 Recaptcha Slashdot 50 Yahoo 0 50 0 50 0 50 0 50 0 50 0 3 4 5 6 7 8 9 10111213141516171819202122232425262728293031323334353637383940 http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Accuracy by education Image captcha Audio captcha 0.8 0.7 0.6 0.88 0.88 0.87 0.87 0.85 0.5 0.4 0.54 0.54 0.52 0.51 0.51 0.3 0.2 Not formal High School Bachelor Master Phd http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Solving time by education 60 Solving time for image Solving time for audio 50 40 seconds 30 20 23.67 23.25 21.33 10 19.75 19.44 9.6 9.36 9.16 8.49 7.64 0 Not formal High School Bachelor Master Phd http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation

Image captcha accuracy time by language Native speakers Non-Native speakers 0.881 Yahoo 0.874 0.946 Blizzard 0.955 0.867 Slashdot 0.890 0.954 Skyrock 0.956 0.738 Recaptcha 0.772 0.804 Microsoft 0.794 0.700 Mail.ru 0.704 0.861 Google 0.873 0.935 eBay 0.935 0.919 Digg 0.925 0.837 captchas.net 0.848 0.927 Baidu 0.928 0.976 Authorize 0.979 0.70 0.75 0.80 0.85 0.90 0.95 1.00 http://ly.tl/p11 E. Bursztein, S. Bethard, C. Fabry, J. Mitchell, D. Jurafsky How Good are Humans at Solving CAPTCHAs? A Large Scale Evaluation