Todai Robot Project Can a machine solve university entrance exam problems automatically? Noriko H. Arai National Institute of Informatics
Todai Robot Project Pursue a real breakthrough by challenging a real intellectual task through the reunion of the AI achievements in the past 30 years Milestones 2016 – Mark a high score in the National Center Test – “Comprehension & Thinking” ・ Computer algebra (Quantifier elimination of RCF problems) ・ Factoid ・ Textual entailment recognition, … 2021 – Pass the entrance exam of the University of Tokyo – “Comprehension, Thinking & Answer generation” ・ Document summarization, ・ Deep and precise language processing, ・ Machine translation, ・ Software component integration framework, …
University entrance exams in Japan National Center Test (multiple choice) 7 subjects • Mathematics (Introductory, Advanced) • Natural Science (Physics, Chemistry, Biology, Earth Science) • Social Studies (World History, Japanese History, Economics & Politics, Ethics, Geography) • Japanese (Contemporary Japanese & Japanese&Chinese Classics ) • English Tokyo University Second Stage Exam (written test) • Mathematics • 2 × Natural Science or 2 × Social Science • Japanese • English
2011
MOZART'S LAST & PERHAPS MOST POWERFUL SYMPHONY SHARES ITS NAME WITH THIS PLANET
MOZART'S LAST SYMPHONY
20 years exam data < Dictionaries Wikipedia JA…
“ A Pendulum Swung Too Far” (Ken Church, ACL-2011) DARPA AI Projects(2010 ～ ) Todai Robot Project （ 2011 ～） : NII Project ARISTO (2013 ～ ) : Allen Institute for AI Integration of Underlying Technologies Modern Hybrid of Logical and Statistical Approaches
Development Technology integration & Improvement • Integration of elemental technologies Technologies for secondary exams • Language understanding boosted by • Text summarization Evaluation domain knowledge and inference • Meta-knowledge structure recognition • co-reference & zero anaphora resolution • Undecidable math problems • Image and NLP • Qualitative reasoning Basic technologies ….. for Center Tests We’re • Syntactic parsing here • now! Textual entailment recognition • Physical simulation platform • Semantic language design • • Development of end-to-end Semantic analysis systems with new technologies Mathematica, • Performance analysis and Watson, Tsubaki, improvement SyNRAC... • Data building • Baseline systems based on • Problem analysis existing technologies • Accuracy analysis Start 2011 2016 2013
Textual Entailment Recognition 2009 Center Test World History B Choose the correct statement about military systems. 1. The Janissaries were standing troops in the Ottoman Empire. 2. The Frankish Kingdom established the thema system. standing troops in the Ottoman Empire X ← ... units that formed X ← Ottoman sultan’s household troops Janissary ... The Janissaries were infantry Musketeer units that formed the Ottoman sultan’s household troops and bodyguards. The force was created by the Sultan Murad I from Christian boys ... Theme (Byzantine district) ... The themes or themata were the main administrative divisions of the middle Byzantine Empire. … Wikipedia
World History Problems via Textural Entailment Recognition Multiple choice problems as textual entailment recognition ⇒ judges truth/falsehood of a text t2 under the premise t1 t1 : Wikipedia & Textbooks Evaluation tasks in NTCIR-11 t2 : Choices in Social Studies Questions Q. Select a correct statement from 1)-3): System Points (/100) 1 ） The Eight Banners was an army founded by the Shunzhi Emperor. Shizuoka U. 57 ○ 2 ） The Janissaries were the standing army of the Ottoman Empire. CMU1 55 3 ） In Francia, a system of farmer-soldiers was established under the CMU2 52 theme system (system of military districts). CMU3 48 YNU 46 ACL 2014 Accurate entailment recognition CMU4 45 “Logical inference on dependency-based by logic/statistics hybrid system compositional semantics” CMU5 43 Fujitsu Lab 41 ・ Expressive & efficient meaning were Fujitsu R&D 37 OBJ SBJ representation by algebraic forms ARG ARG Fujitsu Lab2 34 with set operators the army The Janissaries Hokkkaido U. 31 POSS ・ Inference by logical operation ARG Fujitsu Lab3 23 Ottoman Empire and machine learning Baseline 20
Three Strategies for World History B By combining the three strategies, it became possible to solve the various questions • logical Syntax Tree Matching This area is needed for Strict Strict for the secondary exam detecting the (descriptive) correct choice local global Question Answering Robust for many types Strong for of question Rough detecting the wrong choice Word Co- statistical occurrence
Example ） Using Question Answering • Converting the choice to the factoid question “Charlemagne defeats the Magyar at the 8th century.” (false choice) – – → “Charlemagne defeats (PersonType) at the 8th century.” → ？ Actually, “ Avars ” is Search Results in textbooks and Wikipedia correct … At the end of the 8th century, the Avars that had dominated this land was subsumption to the Frank kingdom under attack of Charlemagne . … （ Wikipedia ） Distance = 14 words Rank Word Score Convert the distance to the score 1 Avars 3.2 2 Mongolian … Calculate the difference from the first place as the cost … … … 5 Magyar 1.1 Cost of “Magyar” is 3.2 − 1.1= 2.1 The score in 2015: 76
How about mathematics?
Let 𝑚 be the trajectory of 𝑢 + 2, 𝑢 + 2, 𝑢 f or 𝑢 ranging over ℝ. 𝑃 0, 0, 0 , 𝐵 2, 1, 0 , and 𝐶 1, 2, 0 are on a sphere, 𝑇, centered at 𝐷 𝑏, 𝑐, 𝑑 . Determine the condition on 𝑏, 𝑐, 𝑑 for which 𝑇 intersects with 𝑚. （ Hokkaido Univ. 2011)
An Image of Automatic Math Problem Solving Let 𝑚 be the trajectory of 𝑢 + 2, 𝑢 + 2, 𝑢 f or 𝑢 ranging over ℝ. Problem 𝑃 0, 0, 0 , 𝐵 2, 1, 0 , and 𝐶 1, 2, 0 are on a sphere, 𝑇, centered at 𝐷 𝑏, 𝑐, 𝑑 . Determine the condition on 𝑏, 𝑐, 𝑑 for which 𝑇 intersects with 𝑚. （ Hokkaido Univ. 2011 ） Machine Translation 𝑏 2 + 𝑐 2 = 𝑠 2 (2 − 𝑏) 2 + 1 − 𝑐 2 = 𝑠 2 (1 − 𝑏) 2 + 2 − 𝑐 2 = 𝑠 2 Logical Form 𝑦 = 𝑢 + 2 𝑧 = 𝑢 + 2 𝑨 = 𝑢 𝑦 2 + 𝑧 2 + 𝑨 2 − 5 3 𝑦 − 5 3 𝑧 − 2𝑑𝑨 = 0 CA & ATP Answer 16
Math - Jointing NLP and CA&ATP through Logic Let 𝑚 be the trajectory of 𝑢 + 2, 𝑢 + 2, 𝑢 f or 𝑢 ranging over ℝ. Problem 𝑃 0, 0, 0 , 𝐵 2, 1, 0 , and 𝐶 1, 2, 0 are on • Syntactic Parsing a sphere, 𝑇, centered at 𝐷 𝑏, 𝑐, 𝑑 . • Discourse analysis Determine the condition on 𝑏, 𝑐, 𝑑 for which Language 𝑇 intersects with 𝑚. （ Hokkaido Univ. 2011 ） Understanding Logical Form in ZF Math Knowledge-base Formula Rewriting • RCF-QE • Gröbner basis Logical Form in RCF etc. Computer Algebra Joint Research with Fujitsu Lab. Answer 17
Is it possible to determine the local theory just from wordings? • Let O be a circle of radius 1 centered on the origin. Given points A and B on the circumference of O , find the point on the x-axis equidistant from A and B. ∊ RCF – • Let O be a circle of radius 1 centered on the origin. Find a point A on the x-axis such that the distance from point A to the origin is equal to the length of the circumference of O . ∉ RCF –
Demo C は原点と (1,1) を通る円である。 C is a circle that passes through the origin and (1, 1). (1) C が x 軸と接するとき、 C の半径を求めよ。 Find the radius of C when C is tangent to the x-axis. (2) C の直径の最小値を求めよ。 Find the minimum diameter of C. (1) (2) 1 1 1 1
Tokyo Univ. prep test (Math, 2013) Num. of Num.of people people 35 45 Mathematics Mathematics 40 (humanities) 30 (sciences) Av. =59.4 Av. =57.4 35 Our system = 61.2 25 30 Our system = 59.4 20 25 20 15 15 10 10 5 5 0 0 Deviation Deviation
2016 Center Mock Test Result ① Japanes Intro- Adv- Engli lish sh Engli lish sh Japanese se World 5 Physics cs e Math Math writing ng Listening ening Histor ory Histor ory subjec ects ts Allot 200 100 100 200 50 100 100 100 950 Average 105.4 45.5 42.8 86.0 24.6 49.4 46.6 45.9 416.4 Todai Robot 90.0 75.0 77.0 80.0 16.0 42.0 55.0 76.0 511.0 45.1 64.0 65.8 48.4 40.5 46.5 54.8 66.5 57.8 T-Score 120.0 100.0 80.0 全国平均 東ロボ 60.0 40.0 20.0 0.0 国語 数学ＩＡ 数学 Ⅱ Ｂ 英語 英語 物理 日本史Ｂ 世界史Ｂ （筆記） （リスニング） Todai Robot marked higher than human examinee in Mathematics (Introductory, Advanced), Japanese History and World History. Need more improvements in Physics, Japanese and English.
2016 Center Mock Test Result ② 9000 8000 We’re here now! 7000 6000 5000 4000 3000 2000 1000 0 174 198 222 247 271 295 320 344 368 393 417 441 465 490 514 538 563 587 611 636 660 684 708 733 757 781 806 830 854 879 900 ～ ～ 30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60 62 64 66 68 70 72 74 76 78 80 82 84 86 88 90 ～
Recommend
More recommend
