Automatic Scoring of Automatic Scoring of Handwritten Essays using Latent Handwritten Essays using Latent Semantic Analysis Semantic Analysis Sargur Srihari, Jim Collins, Rohini Srihari, Pavithra Babu and Harish Srinivasan Center of Excellence for Document Analysis and Recognition (CEDAR) Department of Computer Science and Engineering University at Buffalo, State University of New York 1
Overview of Talk Overview of Talk • Reading/Writing by People/Computers – Importance to Secondary Schools – Role of Computers: Artificial Intelligence – School Assessment Test – Performance Measurement • Technology – Optical Handwriting Recognition (OHR) – Automatic Essay Scoring (AES) – Proposal for an Integrated System 2
3Rs: Computers and Humans 3Rs: Computers and Humans As a goal of As a Human Skill Taught in Schools Artificial Intelligence • • Reading comprehension is necessary for Computers extensively assist (i) academic achievement in all school subjects people in the domain (ii) for economic self-sufficiency in cognitively of doing arithmetic demanding work environments • Writing cannot be imagined without the • Improving reading comprehension will provide use of computers. all members of society with equal opportunities to attain a high level of literacy • Reading by computer is the last frontier: • Writing is the primary means of testing students – Grand challenge of on state assessments AI: read a text-book chapter and answer questions at end • Require appropriate assessment methods computers can help 3
FCAT Sample Test FCAT Sample Test Read, Think and Explain Question (Grade 8) Read the story “The Makings of a Star” before Reading Answer Book answering Numbers 1 through 8 in Answer Book. 4
Why Automatic Assessment Why Automatic Assessment Technologies? Technologies? • Timely scoring and reporting results is difficult • Intense need to test later in school year for – capturing most student growth and – requirement to report scores before summer break • Biggest challenge is reading and scoring handwritten portion of large scale assessment • Automated marking of written text assignments has great value to teachers and educational administrators – When large nos. of assignments are submitted at once, – teachers bogged down to provide consistent evaluations and high quality feedback to students – within short time frame-- in days not weeks 5
Test Modalities Test Modalities • On-Line – Key-boarding skills • How early to introduce? – Computer network down-time – Academic integrity • Paper and Pencil – Natural means of communication 6
Relevant Technologies Relevant Technologies 1. Optical Handwriting Recognition (OHR) • Scanning • Form analysis and removal • Handwriting recognition and interpretation 2. Automatic Essay Scoring (AES) • Latent Semantic Analysis (LSA) 7
OHR: State of the Art OHR: State of the Art • OHR differs from dynamic handwriting recognition – as used in PDAs • OHR System in use by USPS – 90% automatically interpreted • Systems in use for Questioned Document Examination – CEDAR-FOX 8
NY English Language Arts Assessment NY English Language Arts Assessment (ELA)- -Grade 8 Grade 8 (ELA) 9
Sample Question and Answers Sample Question and Answers How was Martha Washington’s role as First Lady different from that of Eleanor Roosevelt? Use information from American First Ladies in your answer. 10
Holistic Rubric Chart for “ “American American Holistic Rubric Chart for First Ladies” ” First Ladies 6 5 4 3 2 1 Understanding of Understanding Logical Partial Readable Brief text roles of first understanding ladies Accurate Not logical Repetitive Understanding of Drawing similarities and Organized conclusions Only literal Limited Understood differences about roles of understanding understanding only sections among the roles first ladies Not thoroughly of article elaborate Characteristics of Sketchy Organized first ladies Weak Too generalized • Complete • Accurate Facts without • Insightful synchronization • Focused • Fluent • engaging 11
OHR using CEDAR system OHR using CEDAR system Scanned Answer Form Removal Line/Word Automatic Word Segmentation Recognition 12
Recognition is based on a Lexicon of Recognition is based on a Lexicon of “American First Ladies American First Ladies” ” “ 1 8 0 0 s an center did fam ily held initial m artha partner role than us 1 8 4 9 and century diplom ats fdr helped inspected m eet people roosevelt that usually 1 9 2 1 anna colum n discussion fdrs her its m iles play roosevelts the very 1 9 3 3 appointed com m unity doing few him jam es m uch polio royalty their vote 1 9 4 5 aristocracy conference dolley first his job nation politicians saw there w ant 1 9 6 2 articles considered during for hom em aking just nations politics schools they w ar 3 8 0 0 0 as contracted early form er honor know n new spaperp residency service this w as a at could ears franklin honored ladies not president sharecroppers those w ashington able be country easily from hospitals lady occasions presidential she to w eakened about becam e create education funeral hostess lecture of presidents should tours w ell across began Curse eleanor garm ent hosting life often press skills travel w ere adlai boys daily elected gathere hum an light on prisons social traveled w hen d after brought darkness encountered husband like opened property society travels w here general allow ed but days equal husbands lim ited opinions proposals som e treated w hich george along by dc established ideas m ade or public states trips w ho girls also call death even ii m adison other quaker stevenson troops w hom given alw ays called decided ever im portant m adisons our rather strong truly w hose great am bassadorcam e declaration everything in m agazines outgoing really students trum an w ife had am erican candidate delano expanded inaugural m ake overseas receptions suggestions tw o w ill half candle delegate eyes influence m aking ow n rem arkable sum m ed united w ith harry depression factfinding influences m any part rights take universa w om an career he l m arried taylor w om ans up w om en w orkers w orld w ould w rote 13 year years zachary
Latent Semantic Analysis Approach Latent Semantic Analysis Approach to AES to AES Human graded documents form training set • Information Retrieval (IR) technique • Holistic characteristics of answer Test document is matched against document graded documents • Useful for document classification • Coarse granularity • Need sample answer documents • No explanatory power, • e.g., principal component value = 30 14
Latent Semantic Analysis (LSA) Latent Semantic Analysis (LSA) • Goal: capture “contextual-usage meaning” from document – Based on Linear Algebra – Used in Text Categorization – Keywords can be absent Document term matrix Projected locations of 10 Answer M (10 x 6) SVD: D o c u m e n t t e r m s Documents in two dimensional plane M = USV T1 T2 T3 T4 T5 T6 S where A1 24 21 9 0 0 3 t Principal Component Direction 2 u S is 6 x 6: A2 32 10 5 0 3 0 d diagonal A3 12 16 5 0 0 0 e New n A4 6 7 2 0 0 0 elements documents t A5 43 31 20 0 3 0 are eigen A6 2 0 0 18 7 16 A values of n A7 0 0 1 32 12 0 for each s A8 3 0 0 22 4 2 w Principal e A9 1 0 0 34 27 25 Component r A1 6 0 0 17 4 23 s direction 0 15 Principal Component Direction 1
Latent Semantic Analysis Latent Semantic Analysis • LSA statistically studies how the variations in term choices and variations in answer document meanings are related. • The simultaneous representation of all the answer documents as points in semantic space 16
Dimensionality of Semantic Space Dimensionality of Semantic Space • Initial dimensionality = number of terms in the document • Dimensionality Reduction – Using SVD – Small enough to facilitate elimination of irrelevant representations – Large enough to represent the structure of the answer documents 17
Singular Value Decomposition Singular Value Decomposition • SVD or two-mode factor analysis decomposes this rectangular matrix into three matrices. M=TSD T – M – is the rectangular term by document matrix with t rows and n columns – T – is the t x m matrix, which describes rows in the matrix M as, left singular vectors of derived orthogonal factor values – D – is the m x n matrix, which describes columns in the matrix M as, right singular vectors of derived orthogonal factor values – S – is the m x m diagonal matrix of singular values such that when, T, S and DT are matrix multiplied M is reconstructed. – m - is the rank of M = min(t , n) 18
Reducing the the Dimensionality Dimensionality Reducing 19
Similarity Measures Similarity Measures 20
Recommend
More recommend
