Towards Focus Detection in Content Assessment Ramon Ziai Towards Focus Detection Introduction Project Background in Content Assessment Empirical Basis IS in Content Assessment Given/New Focus/Background Annotation Ramon Ziai Experiment Data Selection Annotation Criteria Universit¨ at T¨ ubingen Results SFB 833, Project A4 Conclusions References Second T¨ uBerlin Workshop on the Analysis of Learner Language T¨ ubingen, Dec 5, 2011 SFB 833 1 / 27
Towards Focus Outline Detection in Content Assessment Ramon Ziai Introduction Introduction Project Background Project Background Empirical Basis Empirical Basis IS in Content Assessment Given/New IS in Content Assessment Focus/Background Annotation Given/New Experiment Data Selection Focus/Background Annotation Criteria Results Conclusions Annotation Experiment References Data Selection Annotation Criteria Results Conclusions SFB 833 2 / 27
Towards Focus What is Content Assessment? Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content ◮ The task of evaluating an answer to a question with Assessment Given/New respect to meaning given a concrete linguistic context. Focus/Background Annotation ◮ Context here means: Explicit question and reading text Experiment Data Selection in Reading Comprehension data Annotation Criteria Results ◮ Answers to RC questions are produced by language Conclusions learners, language can be ill-formed and requires References robust computational processing SFB 833 3 / 27
Towards Focus Content Assessment: A Simple Example Detection in Content Meurers, Ziai, Ott & Kopp (2011b) Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content Q: Was sind die Kritikpunkte, die Leute ¨ uber Hamburg ¨ außern? Assessment Given/New ‘What are the objections people have about Hamburg?’ Focus/Background - Annotation Experiment TA: Der Gestank von Fisch und Schiffsdiesel an den Kais . Data Selection er The stink of fish and fuel on the quays . Annotation Criteria Results Conclusions Similarity SemType Spelling Spelling Ident Ident References SA: Der Geruch zon Fish und Schiffsdiesel beim Hafen . es- The smell of err fish err and fuel at the port . SFB 833 4 / 27
Towards Focus CREG Detection in Content Corpus of Reading Comprehension Exercises in German Assessment Meurers, Ott & Ziai (2011a) Ramon Ziai Introduction Project Background Empirical Basis IS in Content ◮ Together with Ohio State University and Kansas Assessment University we are collecting a corpus of Given/New Focus/Background ◮ reading texts, Annotation Experiment ◮ questions on the reading texts, Data Selection ◮ student answers to the questions, and Annotation Criteria Results ◮ target answers pre-specified by teachers. Conclusions ◮ All student answers are rated by two annotators at OSU References and KU with respect to meaning. ◮ Binary: correct, incorrect ◮ Detailed: correct answer, missing concept, extra concept, blend, non-answer SFB 833 5 / 27
Towards Focus CREG: Numbers and Sizes Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content Assessment Given/New ◮ Snapshot from September 6, 2011: Focus/Background Annotation ◮ 118 Texts Experiment ◮ 752 Questions Data Selection Annotation Criteria ◮ 1.059 Target Answers Results ◮ 20.851 Student Answers Conclusions References SFB 833 6 / 27
Towards Focus Today: IS of answers to RC questions Detection in Content Assessment Ramon Ziai Introduction ◮ So far, our approach has no notion of information Project Background Empirical Basis required by the question, it only compares student and IS in Content target answers Assessment Given/New ◮ Comparable systems such as C-rater (Leacock & Focus/Background Annotation Chodorow 2003) or Willow (P´ erez Marin 2007) also Experiment Data Selection don’t take the question into account Annotation Criteria Results ◮ However, questions naturally impose IS requirements Conclusions on answers References ◮ Research questions: ◮ What are the necessary IS notions? Given/New or Focus/Background? ◮ Can instances of them be identified reliably? SFB 833 7 / 27
Towards Focus IS in Content Assessment Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content ◮ In order for a system to distinguish between potentially Assessment Given/New relevant and irrelevant content, a partitioning of the Focus/Background answers would be useful Annotation Experiment ◮ The system can only recognize previously mentioned Data Selection Annotation Criteria lexical material so far (simple Givenness) Results Conclusions ◮ This simple case of Givenness is easy to detect, but References does it suffice for the task? SFB 833 8 / 27
Towards Focus Given/New: Example where it works Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content Assessment Welche kulturelle Personlichkeit ist f¨ Question: ur Given/New die Salzburger am wichtigsten? Focus/Background Annotation Target Answer: Mozart [ist f¨ ur die Salzburger am Experiment Data Selection wichtigsten] G . Annotation Criteria Results Student Answer: Mozart [ist die kulturelle Pers¨ onlichkeit Conclusions f¨ ur die Salzburger an wichtigsten] G . References SFB 833 9 / 27
Towards Focus Given/New: Why it works Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content Assessment ◮ No extraneous, irrelevant new information Given/New Focus/Background ◮ [ Mozart ] is the only new piece of information and also Annotation Experiment the correct answer Data Selection Annotation Criteria ➥ When new information is identical to requested Results information, a Given/New distinction works as intended Conclusions References ◮ What happens in other cases? SFB 833 10 / 27
Towards Focus Given/New: Example where it doesn’t work Detection in Content Assessment Ramon Ziai Introduction Question: An was denken viele Menschen, wenn Project Background sie von Weißrussland h¨ Empirical Basis oren? IS in Content Target Answer: [Sie denken an] G die Tschernobyl- Assessment Given/New Katastrophe von 1986. Focus/Background Student Answer: [Ausl¨ ander denken bei Weißrussland] G Annotation Experiment weniger an Urlaub, sondern eher Data Selection Annotation Criteria an die Tschernobyl- Katastrophe von Results 1986. Damals explodierten in der Sow- Conclusions References jetunion Teile eines Atomkraftwerks und wurden einige Regionen Weißrus- slands von der radioaktiven Strahlung verseucht. SFB 833 11 / 27
Towards Focus Given/New: Why it doesn’t work Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis IS in Content Assessment ◮ Lots of new and unrequested information in the student Given/New Focus/Background answer Annotation Experiment ◮ Relevant information [ die Tschernobyl-Katastrophe von Data Selection 1986 ] is included, but cannot be distinguished as such Annotation Criteria Results ➥ Newness is not an ideal category for requested answer Conclusions References content. SFB 833 12 / 27
Towards Focus Given/New: Why it really doesn’t work Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis Extreme case: Alternative questions IS in Content Assessment Question: Ist die Wohnung in einem Neubau oder Given/New Focus/Background einem Altbau? Annotation Experiment Target Answer: [Die Wohnung ist in einem Neubau] G . Data Selection Student Answer: [Die Wohnung ist in einem Neubau] G Annotation Criteria Results Conclusions ➥ Requested information is Given here, so Newness References doesn’t help at all! ◮ What about Focus/Background? SFB 833 13 / 27
Towards Focus What kind of focus? Detection in Content Assessment Ramon Ziai Introduction Project Background Empirical Basis ◮ We need a notion of focus that selects the minimal IS in Content Assessment acceptable answer out of the whole answer content Given/New Focus/Background ◮ Pragmatic notion of focus, no relation to prosody layer Annotation Experiment ◮ Needs to be able to cover phrases, because acceptable Data Selection Annotation Criteria answers rarely consist of single words Results ➥ Similar concept to “focus phrase” in ?? Conclusions References ◮ Needs to correspond to the type of information requested by the question SFB 833 14 / 27
Towards Focus Focus/Background: Example Detection in Content Assessment Ramon Ziai Introduction Question: An was denken viele Menschen, wenn Project Background sie von Weißrussland h¨ Empirical Basis oren? IS in Content Target Answer: Sie denken an [die Tschernobyl- Assessment Given/New Katastrophe von 1986] F . Focus/Background Student Answer: Ausl¨ ander denken bei Weißrussland Annotation Experiment [weniger an Urlaub, sondern eher Data Selection Annotation Criteria an die Tschernobyl- Katastrophe von Results 1986] F . Damals explodierten in Conclusions References der Sowjetunion Teile eines Atom- kraftwerks und wurden einige Regio- nen Weißrusslands von der radioak- tiven Strahlung verseucht. SFB 833 15 / 27
Recommend
More recommend
