ICALL: Part III ICALL: Part III The importance of meaning Diagnosing Diagnosing meaning errors meaning errors Detmar Meurers Detmar Meurers Universit¨ at T¨ ubingen Universit¨ at T¨ ubingen Intelligent Computer-Assisted Language Learning Introduction Introduction Part III: Diagnosing Meaning Errors in ICALL Importance of Meaning Importance of Meaning Existing ICALL / Limitations Existing ICALL / Limitations ◮ Meaningful interaction in the foreign language is an Shifting the perspective Shifting the perspective Exercise Spectrum essential component of second language acquisition. Exercise Spectrum Exercise Properties Exercise Properties Examples ◮ Communicative language teaching, content-based Examples Detmar Meurers The Middle Ground The Middle Ground instruction and task-based language teaching all stress Reading comprehension Reading comprehension (Universit¨ at T¨ ubingen) Our learner corpus Our learner corpus the importance of meaning and exchange of information Gold standard annotation Gold standard annotation Basic idea behind approach Basic idea behind approach in language learning (Richards & Rodgers 2001). CAM CAM based on joint research with Stacey Bailey. See: Bailey/Meurers (2009): General Architecture General Architecture NLP tools ⇒ Meaning (content) assessment is a critical component NLP tools “Diagnosing Meaning Errors in Short Answers to Reading Comprehension”. Alignment Types & Levels Alignment Types & Levels Proceedings of the Third ACL Workshop on Innovative Use of NLP for Building Error Diagnosis Features for intelligent computer-aided language learning Error Diagnosis Features Educational Applications . http://purl.org/dm/papers/bailey-meurers-08.html Results Results Related Work (ICALL) systems in real-life language teaching practice. Related Work Future work Future work Interpretation in Context Interpretation in Context Diagnosis categories Diagnosis categories European Summer School in Language, Logic, and Information Adaptivity (shallow/deep) Adaptivity (shallow/deep) Beyond English Beyond English Bordeaux. July 27–31, 2009 Conclusion Conclusion 1 / 39 2 / 39 ICALL: Part III ICALL: Part III Implications for ICALL activities Implications for ICALL content processing Diagnosing Diagnosing meaning errors meaning errors Detmar Meurers Detmar Meurers Universit¨ at T¨ ubingen Universit¨ at T¨ ubingen Introduction Introduction Importance of Meaning Importance of Meaning Existing ICALL / Limitations An ICALL system that can be effectively integrated into Existing ICALL / Limitations Shifting the perspective Shifting the perspective For an ICALL system to be effectively integrated into different types of language instruction is one that is Exercise Spectrum Exercise Spectrum language instruction, it must Exercise Properties Exercise Properties ◮ Holistic : The ICALL system should process both form Examples Examples The Middle Ground The Middle Ground ◮ offer more than drills and other form-based activities, and meaning of learner responses and, in the latter Reading comprehension Reading comprehension Our learner corpus Our learner corpus ◮ provide a range of contextualized, meaningful language case, extract a representation of meaning, Gold standard annotation Gold standard annotation Basic idea behind approach Basic idea behind approach learning activities, and ◮ Flexible : Processing of learner responses must be CAM CAM General Architecture General Architecture ◮ recognize multiple realizations of the same semantic adaptable, based on the goals of the activity. NLP tools NLP tools Alignment Types & Levels Alignment Types & Levels content in learner responses to an activity. ◮ Robust : The system must analyze meaning even in the Error Diagnosis Features Error Diagnosis Features Results Results presence of form errors. Related Work Related Work Future work Future work Interpretation in Context Interpretation in Context Diagnosis categories Diagnosis categories Adaptivity (shallow/deep) Adaptivity (shallow/deep) Beyond English Beyond English Conclusion Conclusion 3 / 39 4 / 39
ICALL: Part III ICALL: Part III Existing ICALL systems Existing ICALL systems Diagnosing Diagnosing meaning errors meaning errors Background Limitations Detmar Meurers Detmar Meurers Universit¨ at T¨ ubingen Universit¨ at T¨ ubingen ◮ Until recently, research into morphological and Introduction Introduction ◮ Meaning assessment in existing ICALL systems is Importance of Meaning Importance of Meaning structural processing has dominated NLP technology Existing ICALL / Limitations Existing ICALL / Limitations typically accomplished through form comparison. Shifting the perspective Shifting the perspective development. Exercise Spectrum ◮ If the form matches in comparing a learner and target Exercise Spectrum Exercise Properties Exercise Properties ◮ In consequence, most existing ICALL systems have response, the meaning is correct. Examples Examples The Middle Ground The Middle Ground addressed form assessment rather than meaning ◮ This approach is successful due to restrictions on Reading comprehension Reading comprehension Our learner corpus Our learner corpus assessment. Gold standard annotation exercise types in which variation is not expected or Gold standard annotation Basic idea behind approach Basic idea behind approach allowed (Ex: cloze, build-a-sentence, translation). ◮ This emphasis on form assessment has limited the CAM CAM General Architecture General Architecture ◮ This limited processing fails for meaning assessment types of exercises that have been offered in existing NLP tools NLP tools Alignment Types & Levels Alignment Types & Levels whenever variation occurs. For example: ICALL systems. Error Diagnosis Features Error Diagnosis Features Results Results ◮ Character-by-character string matching fails on ◮ German Tutor (Heift 2001) – Uses activities such as Related Work Related Work Future work Future work responses with variation in capitalization or spacing. build-a-sentence that restricts responses to include Interpretation in Context Interpretation in Context Diagnosis categories Diagnosis categories ◮ Token-by-token string matching fails on variation in supplied word forms. Adaptivity (shallow/deep) Adaptivity (shallow/deep) Beyond English Beyond English ◮ BANZAI (Nagata 2002) – Extensively uses translation to spelling, lexical material, word order or structure. Conclusion Conclusion restrict expected responses. 5 / 39 6 / 39 ICALL: Part III ICALL: Part III Shifting the perspective of ICALL system design Relating language exercises and NLP Diagnosing Diagnosing meaning errors meaning errors Detmar Meurers Detmar Meurers ◮ The more variation is possible in learner responses to Universit¨ at T¨ ubingen Universit¨ at T¨ ubingen ◮ Fortunately, the analysis of meaning is increasingly a Introduction an exercise, the more processing is required for Introduction topic addressed by research in computational Importance of Meaning Importance of Meaning meaning assessment. Existing ICALL / Limitations Existing ICALL / Limitations linguistics. Shifting the perspective Shifting the perspective ◮ A spectrum of exercises and meaning analyses falls out Exercise Spectrum Exercise Spectrum ◮ It is possible to focus on what language instructors Exercise Properties of this relationship between exercises and NLP . Exercise Properties Examples Examples need – form or meaning processing – and to allow The Middle Ground The Middle Ground Reading comprehension Reading comprehension language exercises to drive the technology used in Our learner corpus Tightly Restricted Responses Loosely Restricted Responses Our learner corpus Gold standard annotation Gold standard annotation ICALL systems. Basic idea behind approach Basic idea behind approach The Middle Ground CAM CAM ◮ To do this, we need to know General Architecture Decontextualized Short-answer reading Essays on General Architecture grammar fill-in- comprehension individualized NLP tools NLP tools ◮ what existing language learning exercises should be the-blanks questions topics Alignment Types & Levels Alignment Types & Levels Error Diagnosis Features Viable Processing Ground Error Diagnosis Features targeted and what their properties are, Results Results ◮ whether these exercises can be adapted to an ICALL Related Work Related Work Future work Future work system, and ◮ At one extreme, there are restricted exercise types Interpretation in Context Interpretation in Context Diagnosis categories ◮ whether existing NLP technology can effectively Diagnosis categories requiring minimal analysis in order to assess meaning. Adaptivity (shallow/deep) Adaptivity (shallow/deep) Beyond English process the targeted exercise types. ◮ At the other extreme are free-response exercises Beyond English Conclusion Conclusion requiring extensive meaning analysis and world knowledge. 7 / 39 8 / 39
Recommend
More recommend
